The SolarWinds Hack – an IT Security Nightmare

Note: I usually try to write with a business executive in mind as the reader.  While I comprehend and explain what is happening in an article like this, I am not an IT Security specialist and do not intend to present to business leadership the fine details of the SolarWinds hack; that would be a disservice.  But the article will be more technical than I usually write to explain some necessary information and raise the awareness of business leaders when they review their systems with their IT leadership. I also provide more links than usual for a background.

What is the SolarWinds Hack?

A significant IT security incident was discovered in the last few weeks and is still ongoing.  SolarWinds, a software and services company, had malicious code injected into one of their tools and distributed it to their customers.  The hacked software is named “Orion.” This monitors availability and performance for servers, applications, and hardware components.  Because it must see deep into the technology stack, it has administrative access to the domain.  Specifically, the attackers inserted malicious code into SolarWinds.Orion.Core.BusinessLayer.dll, a code library distributed with the installer.

The breach was found on December 13, 2020, by FireEye, an information security firm.  They noticed unexpected activity in their network and determined the cause to be related to Orion, which they used for their network.   We now know it has been active since the Spring of 2020, giving it ample time to do much damage.

The Extent of the Incident

This breach is referred to as “Sunburst” by FireEye and SolarWinds, and “Solarigate” by Microsoft.  It hides inside the installer, which is signed by SolarWinds as a legitimate and safe update so that the operating system will not question its installation.  This approach can be categorized as a “Supply Chain Attack” because it was not an attack directly on one organization; it is injected into the tools they get from their vendors.  The attacker gains access to one system and then have access to all customers who have installed it.

SolarWinds has over 300,000 customers worldwide.  Any of these could be compromised, but not all of these were actively probed.  The attackers filtered their targets to spend time on those who could provide the most valuable information.

As of December 16, 18,000 customers are known to be affected; 80% of these are in the United States.  The victims are from various industries and types.

• 44% were information technology firms, and 18% were government agencies or contractors. Also compromised were healthcare, finance, and telecommunication organizations.
• US government agencies who were hacked include the Treasury, Department of Homeland Security, Department of State, Department of Health’s National Institutes of Health (NIH), the Cybersecurity and Infrastructure Agency (CISA), the National Nuclear Security Administration (NNSA), and the US Department of Energy (DOE).
• Over forty organizations were targeted for more sophisticated measures. In other words, once they had access, they worked more diligently in some high-value targets to expand their exploits.

How does the SolarWinds Hack Work and Hide So well?

As stated above, the SolarWinds hack hides inside software trusted from the supply chain by those installing it. Because of this high trust level, there is not careful inspection or testing, and when running, systems engineers may not question it of anything nefarious.

When operating, it uses numerous methods to reduce visibility. 

• First, it conceals its network traffic as part of the legitimate Orion Improvement Program (OIP) and stores its collected data in authentic files that are part of the system. 
• It also identifies and hides from tools like anti-virus software.  
• It conceals its activity with legitimate log entries; these entries can hide illegitimate activity by data volumes.
• It also sets their hostnames (like a server name) to match a legitimate hostname found within the victim’s environment. Because the log entries were acceptable sources or targets, engineers did not notice it. 

These behaviors hide malicious activity and blend it into the environment to avoid suspicion and evade detection. 

After an initial dormant period of up to two weeks, it begins executing jobs to compromise the systems further and steal data; this includes running programs, gathering more information, and disabling system services.  It can also transfer files.

When it came time to transfer data to a server they controlled, they used the HTTP protocol (like every website).  They further used destination IP addresses located in the same country where the compromised systems reside.  A threat analyst reviewing log files would see regular traffic activity without the usual tip-off of data being transferred to a foreign domain.

Additionally, many organizations lack the skills or tools necessary to identify an ongoing intrusion like this.  It took months for FireEye, an IT security firm, to determine the issue in their network.

Microsoft has a very good graphic of how this works in their blog.

Current Situation

The situation changes quickly, so keeping up to date is critical for an IT team and information security professionals.

SolarWinds has released an updated installer, clean of the malicious code; your systems engineers should install it immediately (see below for a qualifier).

Additionally, Microsoft took control of a key domain the hackers used to control the attack and collect information when they gained access. Commandeering is an essential step in limiting exposure. 

While Microsoft was researching this attack, it found additional malware that also affected the SolarWinds Orion product and determined it to be likely unrelated to this compromise and used by a different attacker.

What Can you do Now to Mitigate the Risks?

There are various steps you should take and do them as soon as possible.

• If you use SolarWinds Orion product, assume you are compromised.  We are not implying you were attacked.  As we said above, the attackers filtered the compromised systems (those with the malicious code installed) for high value.  If you are a small commodity widget manufacturer, it is not likely that you lost anything, but your systems remain insecure, so assume you are not secure.
• Because you are insecure, and the SolarWinds hack is known, it is possible for others who were not the original perpetrators to find the malicious code on your systems and exploit it.  Just because the original actors were looking for extremely high-value targets does not mean someone else would use the unlocked door.
• Decide on the course of action for securing the environment.
  • Decide on what you require in forensic information, if any, for litigation, insurance, or discovery reasons.  If you think you need to maintain the structures’ state as they were for any reason, you will need to backup all systems and preserve them as is.
  • Assume that if the SolarWinds system is patched or built new, the attackers also created other means when they had access.  Once you have a door open, it is easy to go around and unlock some other ones for additional access. You should run a complete network scan and enlist the help of others who are outside information security specialists. A known further breach is called SuperNova, which used holes in the Orion product to install additional malware. In this case, you will need to build all new systems. This approach is obviously costly and will take longer, so consider this wisely.
  • If you will not be maintaining the systems and building new ones, patch the Orion systems to the latest release. More information about this is here.  Also, review all SolarWinds notes here, which is updated almost daily.

• You may also consider the following measures, which are more extensive.
  • Ensure that SolarWinds servers are isolated/contained until a further review and investigation is conducted, and block all Internet egress from SolarWinds servers.
  • If SolarWinds infrastructure is not isolated, consider taking the following steps:
    • Restrict the connectivity scope to endpoints from SolarWinds servers, especially those that are highly-important assets (proprietary information, personal information, etc.)
    • Restrict the number of accounts that have local administrator privileged on SolarWinds servers.
    • Block Internet egress from servers or other endpoints with SolarWinds software.
    • At a minimum, consider changing passwords for accounts that have access to SolarWinds servers/infrastructure.
    • If SolarWinds is used to managed networking infrastructure, consider conducting a review of network device configurations for unexpected and unauthorized modifications.

Summary of the SolarWinds Hack

Solarwinds supplies technology monitoring software used by over 300,000 companies around the world.  Because their source code was hacked, any company using the Orion software is compromised, but most were not maliciously attacked and have lost no data. If you use this product, you should consider your organization compromised and take actions suggested herein and on the SolarWinds website.

In the next article, we will consider what to do in the long run to reduce your risks and plan for even if you do not use SolarWinds Orion.

In the meantime, if you have any questions, contact us for help with this or other IT strategy questions!