CrowdStrike outage shows soft underbelly of connected IT network

One of the key conclusions at the recent GovInsider Live SaaS Day event was that Singapore is actively encouraging the use of cloud-based software-as-a-service (SaaS) applications within government departments in a bid to increase efficiency and digitalisation.

This is a natural outcome of the fact that Singapore is firmly interlinked with globally interconnected IT networks and these networks have provided the country with advantages in terms of workflows and efficiency.

Last Friday (July 19), Singapore, along with the rest of the world, got a stark reminder of the potential downside of this global interconnectedness.

A faulty line of code in a routine product update by cybersecurity company CrowdStrike resulted in what many are calling the largest IT outage in history whose total cost could topple US$1 billion (S$1.35 billion).

The incident is bound to force a serious rethink among government officials responsible for keeping Singapore’s IT infrastructure safe. Since decoupling from these networks is not an option the focus would be, one guesses, on what further steps are required to build more resiliency.

No easy solutions

There are no easy solutions. Emulating China, which was relatively unaffected, and developing local alternatives in cloud computing and enterprise software is not feasible as the market is too small to sustain locally developed solutions and there are a lot of advantages in being part of the global IT ecosystem.

In Singapore, as well as globally, a lot of research goes into trying to understand the modus operandi used in cyber-attacks. Little focus is given to understanding the importance of building more resilience in the tech stack. This needs to change.

The current CrowdStrike incident should be a good wake-up call for CSA and other government agencies to analyse the tech infrastructure used by Singapore and try to ensure that in case of a future outage, either accidental or intentional, Singapore Inc is better positioned to handle it.

8.5 million affected systems

The affected computers were Windows operating system (OS) based. Globally, Microsoft estimates that 8.5 million computers were hit by the outage triggered by the faulty software update.

As big as the number may sound, as a percentage of the total number of devices running Windows globally, it is minuscule. However, the affected computers were all enterprise devices, which is why there was such a major disruption in global networks.

Fortuitously, Singapore escaped relatively unscathed. A statement on July 19 by the Ministry of Digital Development and Information (MDDI) said while government services were not affected, check-in services for airlines, newspaper publishing and carparks were affected.

Elsewhere healthcare services were affected in many countries and thousands of flights globally were cancelled as companies were greeted with the dreaded “blue screen of death” on their Windows-based computers.

As Microsoft wrote in its blog post: “This incident demonstrates the interconnected nature of our broad ecosystem — global cloud providers, software platforms, security vendors and other software vendors, and customers.”

Vulnerability of the IT edifice

If one were to take a step back, it can be quite scary to consider the vulnerability of the entire IT edifice, where a one-line coding error made by an employee in a relatively small cybersecurity company (CrowdStrike) based in Austin, Texas with revenue in the region of US$3 billion and with less than 8,000 employees can cause the biggest global IT outage in history.

What has been less mentioned in the media is the potential domino effect of this outage that can play out in the coming days and months. Taking advantage of the global panic and the desperate bid by companies to find a workaround, a malware campaign targeting CrowdStrike users with a fake hotfix update has been doing the rounds globally, so much so that the Cyber Security Agency of Singapore (CSA) had to issue an advisory warning about this.

Many cybersecurity experts pointed out that the panic and rush to fix the glitch provided a “great mapping exercise for hackers” and pointed them to a potential pathway into company and government tech stacks.

Telling it like it is

While officials in Singapore have been largely circumspect in their reaction to the CrowdStrike incident, the US Cybersecurity and Infrastructure Security Agency's (CISA) Director, Jen Easterly, has been more forthright in stating the obvious.

She wrote in a LinkedIn post: “Our nation’s (America) critical infrastructure, the systems and services that Americans rely on every hour of every day for power, water, transportation, communication, healthcare, education, finance…and much more…is, broadly speaking highly digitised, highly interdependent, highly connected, and highly vulnerable.”

She also noted that “this is due, in large part, to a fragile software ecosystem that has historically deprioritised security in favour of features and speed to market”.

Easterly added that companies like CrowdStrike and other cybersecurity vendors exist to “bolt on security to software that’s been shipped chockfull of vulnerabilities”.

A monopoly situation

There is no doubt that cybersecurity firms play an important role in keeping networks safe, but the industry does have a monopoly-like situation in which just 15 companies account for 62 per cent of the cybersecurity market globally. So, if one of them makes a misstep like CrowdStrike, there is a widespread repercussion.

In 2010, a similar botch up with an update by McAfee brought down thousands of computers. Ironically George Kurtz, the current CEO of CrowdStrike was the CEO of McAfee when the incident happened.

Another fun fact: Just six companies, including Microsoft, CrowdStrike and TrendMicro, control 44.2 per cent of the global endpoint security market.

Many security companies run daily updates or patches, like the one sent out by CrowdStrike, to keep their software updated to tackle the latest threats.

The way forward should be a full analysis to identify potential vulnerabilities. Along with that, there should be outreach to major tech vendors to solicit their cooperation in developing both the strategy as well as hard infrastructure to better safeguard local networks.