December 12, 2020 was a black spot on the calendar for IT management firm Solar Winds and its security boss, Tim Brown.
It was the day they found out hackers had modified the company’s core product to send malicious updates to as many as 18,000 of Solar Winds’ global customers.
“People question did we know before then,” Brown told ITWire. “Absolutely not. We were shocked and dismayed on December 12.”
It was supposed to be a routine software update. Fixes for bugs, improvements to performance. But the code shipped to users of the Orion IT monitoring platform for a few months mid last year instead gave hackers secret access to any Orion customers who applied the update and were connected to the internet.
The world was about to discover that something had gone very, very wrong.
Two phones, one person
The first sign that things were not quite right could have easily slipped by unnoticed. A security worker at the cyber incident response firm FireEye spotted that an employee had two phones registered to the company’s network.
For many organisations, this discovery would seem unremarkable and pass by without concern. But for FireEye – a firm that helps investigate and clean up some of the world’s biggest breaches – it was a big red flag.
The security worker called the employee, and quickly realised something malevolent was afoot. Little did they know, it was just the beginning.
Over the next few weeks, FireEye discovered that someone had managed to infiltrate its network and steal the collection of hacking tools FireEye uses to test the security of its customers’ IT environments.
The FBI was called in, and the source of the breach was identified as the Orion platform meant to be keeping tabs on how FireEye’s IT equipment was performing.
From there, things spiralled and it fast became evident just how widespread the problem was.
A subsequent investigation uncovered that hackers had compromised the software update mechanism for Solar Winds’ Orion platform from as early as March 2020 – meaning they’d potentially been in customer networks for as long as nine months.
The US government’s cyber security agency told departments to immediately rip out Solar Winds tech from their networks. High-profile Solar Winds customers – the firm counted the majority of Fortune 500 companies amongst its user base – scrambled to quantify the impact to their operations.
The victim count steadily grew: US government agencies like the Treasury, State Department, Pentagon, Homeland Security, and Department of Energy were compromised.
So too were private companies like technology giants Microsoft , Intel, Cisco, Nvidia, VMware and Deloitte, not to mention hospitals, universities and other important service providers like critical infrastructure.
But the hackers were stealthy, and to this day it’s still not entirely clear what was actually stolen, nor what they were seeking.
“The craziest f***ing thing I’d ever seen”
The origin of the attack was later traced back to September 2019, when the hackers first used a tiny piece of code to test whether they could infiltrate Solar Winds’ software update mechanism and get their malicious files installed in customer networks.
The code itself was unremarkable. But it gave the hackers confidence they could pull off a far-reaching supply chain attack, CrowdStrike threat intelligence chief Adam Meyers told NPR.
“They [now] know that they have that capability.”
But then, silence: the hackers went dark for five months.
When they resurfaced in February 2020, however, they came ready for action.
This time, the hackers had figured out a way to alert themselves to whenever the Solar Winds development team was preparing to build new software, according to NPR.
They managed to gain access to the company’s build system by compromising a service, or robot, account using a combination of techniques: a targeted spear phishing email that successfully captured the targeted person’s login credentials, followed by exploitation of an unidentified vulnerability.
They’d wait until new Orion code was compiling, then at the last minute swap in their malicious update file to replace the legitimate one. They made sure to make the switch just before the update went from source code to executable code – right before it was pushed to customers – to avoid detection.
Their evasion efforts were comprehensive: they made their traffic look like legitimate Orion system communications, and they’d wait two weeks before entering through the backdoor they’d implanted on customer networks so they wouldn’t be spotted.
“There was intelligence in the code so it would only run in certain places. It wouldn’t run in the Solar Winds domain or certain address spaces owned by Microsoft. The code was an entry point to try and move laterally,” Brown said.
“The code was unique in the way it tried to evade things, and where it would run and not run. It inserted code but didn’t weaponise or take action until time had passed. It was smart, patient, and prescriptive, and made very little noise in our back-end environment.”
At just 3500 lines long, their “elegant and innovative” piece of malware – since dubbed “Sunburst” – was so carefully constructed it was almost impossible to definitively prove who wrote it.
“The tradecraft was phenomenal,” Meyers said.
“This was the craziest f***ing thing I’d ever seen.”
Who was impacted?
The hackers appear to have been highly selective about who they went after and what they sought to steal.
In January, FireEye revealed two groups of people were specifically targeted: systems administrators and anyone with access to high-level information. Similarly, it appears government organisations and software companies were prioritised as targets.
In some cases, but not all, the hackers used their access to a victim’s on-premise network to pivot into the organisation’s cloud-based Microsoft Office 365 environment and the sensitive data contained within.
In others, victims were also used as stepping stones into the networks of their own customers – meaning a number of organisations found themselves compromised by the Solar Winds hackers despite having never even used the company’s products.
But actually figuring out to what extent a victim organisation was compromised is difficult – though not impossible – due to the hackers’ sophisticated anti-detection techniques, FireEye said.
However, the goal broadly appears to have been to scour through an organisation’s network and steal any data the hackers considered valuable: we know proprietary source code was viewed. Sealed court records were accessed. Email inboxes were rifled through.
Whether the hackers also used the opportunity to plant something more destructive for future use remains unclear.
The US government said it considers the campaign to be “an intelligence gathering effort“.
To date, at least nine government agencies and as many 100 private sector companies were successfully compromised by the Solar Winds hacker group.
Unmasking the culprits
US, UK and Canadian authorities have publicly attributed the campaign to Russia.
They believe the Russian government’s intelligence service, the SVR, ordered its hacking collective – known amongst other names as APT29 and Cozy Bear – to hijack the Orion update mechanism in order to conduct an audacious and widespread cyber attack against American organisations.
The US government has imposed sanctions on Russia in response and says it is “highly confident” in its attribution of the Solar Winds attack.
“The scope of this compromise is a national security and public safety concern,” it said last month.
“We will continue to hold Russia accountable for its malicious cyber activities, such as the Solar Winds incident, by using all available policy and authorities.”
Russia, for its part, has denied any involvement and called the attribution “nonsense”.
The head of the SVR, Sergei Naryshkin, last month claimed there was no proof his agency had anything to do with the attack, and spuriously argued it was more likely the US and UK were to blame.
“I’d be flattered to hear such an assessment of the work of the foreign intelligence service which I run. Such a high evaluation,” he told the BBC.
“But I don’t have the right to claim the creative achievements of others as my own.”
Meanwhile, Solar Winds has brought in two respected cyber security experts – former government cyber chief Chris Krebs and former Facebook security boss Alex Stamos – to help clean up the mess.
The pair told the Financial Times it will likely take years to remediate the security threat completely.
At the same time, the US government is attempting to raise the bar for those who supply technology to the public service.
A new executive order seeks to modernise the government’s approach to cyber security by, among other things, mandating agency use of multi-factor authentication, strong end-point controls, encryption, zero-trust architectures, mandatory breach reporting, and standardised incident response plans.
Software suppliers to government will also be subject to an “energy star”-type labelling system that rates the secure development of their code.
The Solar Winds attack was a “sobering reminder” of the increasingly sophisticated malicious cyber activity facing government and private sector organisations, US President Joe Biden said.
Others have similarly labelled the breach a strong “wake-up call” for society at large.
“Courses and books will be written because of this,” Brown said.
“It’s one of the most effective cyber-espionage campaigns of all time,” Stamos told NPR.
“This certainly is going to change the way that large enterprises think about the software they install and think about how they handle updates.”
