Most Tools Failed to Detect the SolarWinds Malware. Those That Did Failed Too
The specter of advanced persistent threats has helped sell a lot of cybersecurity technology over the last decade. APTs, as they are known in the trade, are all over the marketing campaigns of every major cybersecurity vendor. And yet, apparently, the actors behind the SolarWinds hack easily evaded them all. For endpoint detection and response (EDR), the threat actor seems to have tested its malware against all the major players. It knew which ones could detect it, which ones it could turn off, and which ones it could not evade.
While we could never know how many companies were actively compromised by the campaign, the list keeps growing. It’s therefore safe to assume that there were many opportunities for detection and other categories of tools, including extended detection and response, automated threat hunting platforms, and internal network monitoring tools, were all evaded for over seven months.
What finally led to the discovery of the intruder at FireEye was not any detection system but some good old fashioned detective work by a system administrator who investigated a failed attempt to add a device for multifactor authentication. For those steeped in the lore of the cybersecurity industry, the parallel to the Cuckoo’s Egg is all too disturbing thirty years later.
While the failure of the cybersecurity industry to detect the campaign after years of relentlessly hyping their capabilities against these actors is troubling, what is even more concerning is that at least one vendor is claiming that they detected and stopped the campaign. In a blog post, Palo Alto Networks, in a bit of a humble brag, noted that they had detected the activity on their own network, thwarted the attack, distributed signatures to protect their customers, but had not realized that it would turn out to be a big deal.
Herein lies why we have spent twenty years talking about information sharing in the field of cybersecurity and why we are doomed to spend another twenty years discussing it. We can’t get it right. In theory, the advantage of the attacker in cyberspace can be reversed if every target they touch is part of one big detection grid. If multiple companies are using different tool sets to hunt for threats along the cyber kill chain, a detection by any one company will unravel the whole campaign.
With this kind of system in place, an attacker would not take the risk of targeting multiple organizations using the same malware, delivered in the same way, using the same command and control channel because the wider they go, the more likely they are to be detected. That is ultimately what happened in the case of SolarWinds, but it took nine months, rather than minutes.
The failure of some or even most detection technologies and threat hunting teams is to be expected—the Russians really are that good. Where I am disappointed is that Palo Alto Networks failed to share their discovery with the broader security community. Beyond Palo Alto, we don’t know what other companies also detected and stopped the intrusion, but I will guess that other companies did as well.
It was Palo Alto Networks, after all, that established the Cyber Threat Alliance to address this very problem. The notion behind the Cyber Threat Alliance was that member companies would share malware samples and other indicators of compromise that they captured to help secure the entire ecosystem.
This is not a condemnation of the Cyber Threat Alliance, of course, whose data is only as good as what its members share. But it is a recognition that even the best-intentioned companies that have pledged to work cooperatively are not doing so in practice, at least when it counts. While we don’t have a complete picture of the campaign and the opportunities that were missed to connect the dots, the picture that is emerging is not a pretty one.
It could be that partnerships like the Cyber Threat Alliance or government efforts like the Cybersecurity and Infrastructure Security Agency Automated Indicator Sharing program should no longer be voluntary. There could be other problems that have inhibited sharing and other potential enhancements. This is why we so badly need a thorough and ongoing investigation of SolarWinds campaign and others that will come.