We conducted tests using real-world malicious domains to evaluate the efficacy of HYAS Protect, along with three other top protective DNS providers
It’s well-known in cybersecurity circles that most malicious traffic employs the domain name system (DNS) in some way. For instance, command and control (C2) infrastructure and malware are routinely hosted on domains owned by threat actors. In fact, The DNS Abuse Institute identifies five broad categories in which DNS is commonly used for malicious purposes. This makes the DNS layer an ideal starting point for enforcement and/or detection. By preventing DNS resolution to malicious domains, malware, and botnets, organizations have the power to stop bad actors in their tracks.
Of course, the bad actors know that we know this (part of the never ending game of cat-and-mouse), so they avoid reusing domains, since a domain is easy to block once it has been identified as malicious.
The ability to reliably block known bad domains is a cornerstone feature of any protective DNS solution, and most providers depend heavily on these feeds to protect their customers. These lists are usually compiled from a number of sources, including open source intelligence, data purchased from third parties (commercial feeds), and in-house research.
However, relying solely on a list only comprising domains that have previously shown malicious traffic is not enough. It cannot protect you when a threat actor uses a new domain or a domain that has not yet been flagged as malicious. In business, being first usually is a good thing, but when it comes to malware, being among the first to be affected is a very bad thing.
Most protective DNS solutions also employ other methods beyond simple block lists to try to identify unknown malicious domains, such as blocking newly registered domains (usually within the last 30 days) or using AI/ML to detect domain generation algorithms (DGA) domains or detecting anomalies in DNS traffic.
While these techniques do improve the catch rate of malicious domains, threat actors are wising-up and adapting their methods to bypass these additional protections. A very common technique now is to let domains age (also called ripening) before they are used so they don’t get flagged as a new domain.
Though HYAS uses the same enforcement mechanism as traditional protective DNS vendors by blocking DNS queries to malicious domains, HYAS does not limit itself to known indicators or common algorithms and anomalies. More importantly, HYAS's approach includes mapping and understanding threat actor infrastructure before the threat actor uses that infrastructure in the wild. This ability, along with HYAS's unrivaled domain intelligence, enables HYAS to block malicious traffic to that infrastructure the first time it gets used.
Of course, actions speak much louder than words, so we conducted a test to verify our efficacy as well as evaluate the efficacy of three other well-known protective DNS solution vendors.
Products Tested
- HYAS Protect
- DNSFilter (Enterprise version)
- Cisco Umbrella DNS Security Advantage
- Quad9
Test Domains
The threat intelligence team at HYAS had been doing some investigations that initially looked at passive DNS for specific indicators, which created a date fence based on when passive DNS saw the FQDN resolve to an IPv4 address. Using this data and referencing VirusTotal within that time frame, the team graphed the IPv4 address that had file communications to it. Then the file communications were analyzed to its URIs and stripped to just the domain.
In short, this generated a list of domains that real live malware was communicating with.
The threat intelligence team went further than this, but for efficacy testing purposes, this is all the data we needed. This produced a list of almost 20,000 domains. Because even malware has some “legitimate” communications — such as talking to CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol) FQDNs — we did some scrubbing and ended up with a final list of 19,578 domains.
Note that this list of domains was not added to any HYAS product before the testing to ensure that it didn’t have an unfair advantage and could only use the knowledge we already had about malicious domains and threat actor infrastructure.
To reiterate, these are 19,578 domains that actual malware was talking to without any legitimate reason.
Test Methodology
HYAS, DNSFilter, and Cisco Umbrella were all set to their most stringent security configurations, blocking all malicious traffic and newly registered domains. Uncategorized domains were not blocked. Although blocking uncategorized domains would increase all vendors’ numbers, it would also result in too many false positives (and frustrated users) to be useful in most real-world environments.
For Quad9, no configuration is possible as this is not a service targeted at enterprise users, but is still included here nonetheless for reference points.
A script would take each domain from the list of 19,578 domains and do an actual DNS lookup with each of the protective DNS solution providers, as well as do an API call to VirusTotal. The results were tallied as follows:
- If the service returned an IP address of a block page, the DNS request was considered blocked.
- If the service returned an NXDOMAIN, the DNS request was considered blocked.
- If the service timed out, the DNS request was considered blocked (note that not a single DNS request to any of the providers was timed out, but the script had provisions for detecting this).
- If the service returned an IP address that was not of a block page, the DNS request was considered allowed.
VirusTotal aggregates information from 94 different security vendors to determine a domain’s risk score. A vendor can flag a domain as:
- Malicious
- Suspicious
- Harmless
- Undetected
- Timeout
For the VirusTotal score of a domain, we combined the number for “Malicious” and “Suspicious”. For example, if a domain gets flagged by three vendors as “Malicious” and two vendors as “Suspicious”, its VirusTotal Score would be five.
If the VirusTotal score is zero, none of the 94 vendors VirusTotal uses flagged the domain as “Malicious” or “Suspicious”.
Results
After running the 19,578 domains through each protective DNS solution, these are the outcomes:
No single security solution will be able to block all malicious traffic, and the results for DNSFilter, Cisco Umbrella, and Quad9 are very similar. However, HYAS Protect blocked many more domains than its competitors.
Filtering this further, there were 10,624 FQDNs that HYAS Protect blocked that got by DNSFilter, Cisco Umbrella, and Quad9. The VirusTotal scores for these 10,624 domains is listed in the table below.
This shows that 2,008 domains were already flagged by at least one vendor as malicious or suspicious but still weren’t caught by any vendor in this test except for HYAS.
The expectation is that the VirusTotal scores, as well as the catch rate of DNSFilter, Cisco Umbrella, and Quad9 for these 10,624 domains, will increase over time. In a follow-up blog, we will show how much these increased.
Conclusion
Having a list or database of known malicious domains is a valuable tool for a protective DNS service and is able to catch a lot of bad traffic. But that in itself is not sufficient, especially for new(er) malware, campaign techniques, and other threats that use never before seen domains. By having superior domain intelligence and mapping of threat actor infrastructure, HYAS is able to provide a much higher catch rate, especially for domains that are not yet known to be malicious — providing demonstrable proof of proactive protection.