Inserting malicious code in open-source libraries is about as easy as reading this text. © wernerwerke

“Sunburst” was the most sophisticated hack the world has ever seen. One of the most urgent lessons so far learned from the attack: we need to enforce global software supply chain management now. ¹

The first details on the “Sunburst” attack were released in December 2020: a highly evasive attacker leveraged the supply chain of the U.S. based software company SolarWinds in order to compromise an administrative I.T. management tool which is used by tens of thousands of private and government organizations. The backdoors and malware that were installed on the victims’ infrastructures have been called by Microsoft “the most sophisticated and protracted intrusion attacks of the decade”.² (That was a week before hundred of thousands of Microsoft exchange Servers were hacked, but alas).

It is still too early to draw definitive conclusions from the first forensic analyses, but this much is clear: supply chain attacks are not just devastating for individual organizations but they fundamentally undermine trust in information technology.

One of the underlying reasons the attack was so successful is that our society, at large, has a fundamental software problem. The issue is systemic, and it is on the verge of turning into an apocalypse.

We are increasingly dependent on computers and the software they run on. In the financial sector, in healthcare, in the military, in the manufacturing industry, and in the service domain, we rely on software we believe can be trusted.

And software is no longer confined merely to computers. It now controls the operation of phones, complex power generators, medical hardware, cars, doorbells, cameras, and kitchen appliances. As software engineer and venture capitalist Marc Andreessen put it already ten years ago, “Software is eating the world.”

The world runs on software. It runs on information technology. But it can’t run with confidence if major governments are disrupting and attacking the software supply chain in this way.
Brad Smith, President of Microsoft.³

Software Supply Chain Attack

The attack on SolarWinds has made it more apparent than ever before that software supply chain attacks are incredibly destructive. Damage done in a tiny module delivered by a small software vendor can break a global software empire’s security. This can lead to a loss of sensitive customer information, disruption of manufacturing processes or reputational damage.

However, there is an even more worrying effect: if attacks are possible at such a scale that their effect is felt across whole sectors, countries or even globally, as is the case with the “SolarWinds” attack, they have the potential to fundamentally undermine our trust in information technology.

Attacking the global software supply chain is easy. It does not take the coordinated effort of thousands of engineers in a nation-state to stage a global supply chain attack. It is straightforward to insert malicious code in open-source libraries. As recently as February 2021, a young hacker described how he set up a successful supply chain attack against companies such as Apple and PayPal by attacking the way common programming languages, such as Python and Ruby, load external modules. The mechanism can be simply tricked into installing compromised modules from public code repositories. ⁴

Supply chain attacks are not new, and they can occur in any industry. Yet, a few risks unique to the software domain make it especially vulnerable to tampering.

The more features a software product has, the larger the potential of bad surprises. Seven in 10 applications use at least one open source library with a security flaw. © wernerwerke

First, all software comes to market with flaws and vulnerabilities that are gradually discovered or reported by users. Then they are corrected by the vendor, at which stage new errors may be introduced. This causes a long tail of patch management in the supply chain. Most applications consist of hundreds or even thousands of functional modules. All of these modules have different modes of responsibility and accountability for the patch process. The more features a software product has, the larger the potential attack surface.

The legal setting may be very different for each of the modules; some are open-source, others are proprietary. That means that it can be hard to figure out responsibilities. Patching is not as straight forward as forcing a subcontractor into action with legal arguments.

Second, bugs, inadvertent or deliberate, can affect all hierarchical layers of a software product. A bug may exist at the lowest level of on-chip software. Then its effect may go to the operating system level. From there, it can affect the data transport level or the application framework level. Then, at the top of the software pyramid, the actual application may be affected.

All these levels have dependencies: an attack on a low system level may be able to corrupt or circumvent protective measures on higher levels. For example, an attack on the operating system level may outsmart all data encryption on the application level.

Third, protocols — technical standards that describe how data needs to be stored, transported, or processed — are not one-and-done safeguards. They can have theoretical limitations, for instance, by flaws in the design — e.g. requiring low-security standards. Or they may be subject to something akin to the decay half-life of a radioactive element, as is the case with encryption standards for which it is a matter of time before computers become powerful enough to break them.

Fourth, software patches for components in different hierarchical levels have different periodicity. On-chip firmware may be updated every five years or so. A typical framework or platform, such as Oracle or SAP, updates quarterly. Industrial applications may have release plans of a few years. Microsoft introduced the monthly “Patch Tuesday” in 2003. Google now also updates its web browser Chrome on a monthly basis. Ad-hoc patches are published in the case of especially critical vulnerabilities.

All of these updates have functional dependencies. Updating components of an on-chip module or an underlying data communication process in an operating system may break things elsewhere. Users need to address the problem by installing newer versions of the dependent module. This, in turn, may break other dependencies and push the issue to another set of packages.

There is a high level of uncertainty in managing the process. On the one hand, the consequences of choosing not to perform an update are reasonably well known: they lead to exposure of at least one known vulnerability. Yet, what is the likelihood that this vulnerability will be exploited?

On the other hand, the consequences of choosing to implement the patch are almost impossible to predict. Dependencies may be known for a specific version of the software on a system but may vary hugely depending on individual configurations or even on the process state of a particular system. Functionalities may break everywhere — in a particular application, in the operating system, between software modules within one system or between interconnected systems. The result is dependency hell.

Bugs affect all hierarchical layers of the software pyramid. © wernerwerke

Dependency hell makes software chains particularly vulnerable to attacks because the software update process requires careful planning and testing, and this can cause extra delays. One of the most significant data breaches in history, Equifax’s 2017 leakage of hundreds of millions of customer records, was possible because the organization failed to update a vulnerability in a low-level web server module on time — meaning within a few months.⁵ It wasn’t a particularly bad or unusual bug, and it would have been routine work to patch. Yet, for whatever reason — perhaps figuring out how to do it without affecting the availability of their systems — it took Equifax three months to do it. That gave malicious actors enough time.

This leads to a fifth reason why the software supply chain is so vulnerable to attacks: the sheer amount of known software vulnerabilities. As of March 1, there were 203.241 vulnerabilities documented in the U.S. government repository of standards vulnerability management database.⁶ For more than a decade, the number of reported vulnerabilities was fairly stable at around 400 per month. But in 2017, the number of reported vulnerabilities doubled, and growth has been exponential since then. This means that secure operations teams get swamped by vulnerabilities every day.

Over the last 20 years, the amount of software vulnerabilities has increased dramatically. As of March 1 2021, there were 203.241 vulnerabilities documented in the U.S. government repository of standards vulnerability management database. Data source: https://nvd.nist.gov © Dr. Rispens, wernerwerke

Attackers have automated scanning and intrusion. Which is another reason for extra vulnerability of the software supply chain: attackers have a strategic advantage, because what they need to do to be successful is much easier automated than what defenders need to do. And of course, they only need to be right once, while defenders need to be right all the time.

A final factor for the extraordinary vulnerability of the software domain to supply chain attacks is that including external software components is applied on a massive scale. A 2017 audit and analysis of over a thousand commercial, cross-sector codebases found that 96% of all software products included third-party software components and commercial off-the-shelf components, as well as modules and libraries from both open-source and commercial third-parties.⁷

The whole assumption behind this is that we can build established channels of system verification to gain privileged access to systems and ICT infrastructures. But this is no longer true: the channels are substantially affected by trends within the software domain, such as globalization, decentralization, outsourcing of the software manufacturing process and now also by nation state actors, who are determined to undermine all chains of trust. Add to that outsourcing ICT management and services to cloud computing and managed service providers, and the already massive supply chain problems in the software domain are multiplied by several orders of magnitude.

All in all, software is essentially a sitting duck for supply chain attacks. What can we do?