In a previous blog, we shared the results of a comparison test that pitted HYAS Protect against three other highly regarded protective DNS providers. To ensure the test’s legitimacy, we used a list of 19,578 domains that we knew were associated with malware or otherwise known to be malicious. Our list initially comprised 20,000 domains, but we took care to weed out any legitimate destinations like CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol) FQDNs before running the test — with 19,578 domains making the cut.
The results of this test demonstrated conclusively that HYAS’s knowledge of threat actor infrastructure allowed the HYAS Cloud to block substantially more malicious domains than the legacy vendors.
A quick reminder: 
After a month had passed, what would happen if we ran the same list through all the vendors again? Had they correctly changed their rating of these domains during the intervening time? In other words, how long does it take for the rest of the world to catch up to what HYAS knows?
Test Methodology
HYAS, DNSFilter, and Cisco Umbrella were all set to their most stringent security configurations, blocking all malicious traffic and newly registered domains. Uncategorized domains were not blocked. Although blocking uncategorized domains would increase all vendors’ numbers, it would also result in too many false positives (and frustrated users) to be useful in most real-world environments.
For Quad9, no configuration is possible as the service is not targeted at enterprise users, but it is included here nonetheless as a reference point.
A script would take each domain from the list of 19,578 and do an actual DNS lookup with each of the protective DNS solution providers, while also making an API call to VirusTotal. We then categorized the results based on the following criteria:
- If the service returned an IP address of a block page, the DNS request was considered blocked.
- If the service returned an NXDOMAIN, the DNS request was considered blocked.
- If the service timed out, the DNS request was considered blocked. (Note that not a single DNS request to any of the providers timed out, but the script had provisions for detecting this.)
- If the service returned an IP address that was not a block page, the DNS request was considered allowed.
VirusTotal aggregates information from 94 different security vendors to determine a domain’s risk score. Any of these vendors can flag a domain as:
- Malicious
- Suspicious
- Harmless
- Undetected
- Timeout
For our test, the “VirusTotal Score” of a domain was obtained by combining the number of “Malicious” and “Suspicious” flags it received in VirusTotal. For example, if a domain gets flagged by three vendors as “Malicious” and two vendors as “Suspicious,” its VirusTotal Score is five. If the VirusTotal score is zero, none of the 94 vendors flagged the domain as “Malicious” or “Suspicious.”
Results
Below are the results for HYAS and three other leading protective DNS solutions one month after our original test. Next to each solution's final total, we have listed the amount of increase or decrease since the last test, allowing us to see to what degree their efficacy changed over time.
Unsurprisingly, all four protective DNS solutions blocked a greater portion of the malicious domains this time around. DNSFilter had the biggest improvement, going from blocking 39 percent of the malicious domains to blocking 54.5 percent. However, that number still pales in comparison to HYAS’s 95.4 percent block rate.
As previously noted, in this test we define “blocked” as either presenting a block page or returning a NXDOMAIN (unable to resolve the query). Quad9 always returns a NXDOMAIN if its verdict for a domain is malicious, however HYAS, DNSFilter, and Cisco Umbrella only return a NXDOMAIN if they cannot resolve a domain. In the initial test there were 2,701 domains that all vendors flagged as NXDOMAIN. In the new test, the number increased to 3,125, most likely indicating that a significant number of these domains were taken down.
When we ran the test the first time, we also ran VirusTotal against a subset of the domains. The subset was constructed by picking the domains that HYAS Protect blocked, but all three of the others allowed, which left us with 10,624 domains in total. For this latest test, we used the exact same subset to find out what changes had taken place on VirusTotal.
On the first test, VirusTotal already flagged 2,008 domains as malicious or suspicious. That has now increased to 2,743, showing that more domains are being discovered and flagged by VirusTotal’s 94 vendors.
Conclusion
Given that malware beacons out to command and control (C2) for instructions, knowing and understanding threat actor infrastructure is an important part of keeping your network safe. Being able to block or alert you on traffic going to malicious destinations long before those destinations are well-known drastically cuts down on the progress and damage bad actors can do. According to the latest data, malware spends a median time of 21 days on a network before being discovered — often only being noticed because the malware has successfully executed its goal. That said, some varieties are able to spend up to a year or more on a network before detection.
The results of our testing has shown that HYAS is capable of detecting malicious domains more than 30 days before the competition — That huge head start that can mean the difference between administrators blocking or containing the malware versus having an attack move into its final, most destructive stage. Taken together, our two tests show that even though other vendors do start to identify bad destinations over time if they continue to be used, waiting for them to catch up or relying on static indicators could put you at risk. Thanks to HYAS’s superior domain intelligence and mapping of threat actor infrastructure, you can catch threats earlier and more reliably than with legacy protective DNS vendors, giving you the advantage you need to keep your business moving forward at full speed.