Which suppliers could you not work without for a day?

· · Infrastructure Security

Most of what your business does runs through suppliers you chose: your bank, your payroll bureau, your line-of-business software, your website host. Each of those runs on suppliers you never chose and probably cannot name. You only find out where those layers meet when one of them stops.

This week, two of them stopped. On Tuesday a power failure in a single Google data centre in London disrupted UK banking apps and the courts' remote hearings. Over the weekend before it, an attack took Bullhorn, the recruitment platform behind thousands of staffing agencies, offline for days. Neither was a system any of those businesses ran themselves. That is exactly why it is worth an hour of your time.

One building, one morning

Google's London region, europe-west2, is split into three "availability zones": separate facilities, so that losing one should not take down the rest. It is Google's only cloud region in the UK.

According to Google's preliminary incident report, published today, mains power to the building housing zone europe-west2-b was interrupted at 06:53 on 20 October. Batteries took the load while the diesel generators were meant to start. They never did: both the primary and the backup control systems failed to trigger them. When the batteries ran out at 07:00, every server, network switch, and storage array in that zone went dark. Google puts the incident at 6 hours and 10 minutes, ending at 13:10, with a "very small number" of customers affected for longer.

The knock-on effects were wide. Revolut customers could not make card payments, withdraw cash, or transfer money for several hours. Lloyds Banking Group and HSBC customers reported login problems, and the video platform used for remote court hearings across England and Wales went down. The Financial Conduct Authority and the Prudential Regulation Authority have asked Google for more information, because Google is now one of the Critical Third Parties to UK finance.

"Spread across zones" did not save everyone

The detail worth reading twice is in Google's own report. Customers who had done the sensible thing and spread their systems across all three London zones were hit too.

When zone B vanished, everything that had been running there tried to restart in zones A and C at once. That surge used up the spare capacity, so databases took longer to fail over, jobs queued, and services returned errors. Some of Google's own management systems in the healthy zones also kept waiting on checks from the zone that was offline. Google names BigQuery, Looker, Compute Engine, Kubernetes Engine, regional persistent disks, and Cloud SQL among the services that degraded in zones that never lost power.

If you build on a public cloud, that is the lesson to take away. A multi-zone design protects you against a lost zone only if there is room to land in the others, and only if nothing you rely on quietly depends on the zone that failed. It is worth checking whether yours has been tested that way, rather than assumed. For many businesses a few hours of degraded service is an acceptable risk. Just make it a decision rather than a surprise.

A week without the system that pays your temps

Bullhorn's problems began on Friday 16 October, with job adverts failing to post. By Saturday the whole platform was down, and the company confirmed the cause: "We have detected a cyber incident on the Bullhorn Platform." It switched off access to contain it and brought in CrowdStrike to investigate. Services began returning in the UK, US, and Australia on Monday, but The Register reported that agencies had lost access to candidate records, placements, and in some cases the systems used to get people paid.

Bullhorn's customer FAQ, last updated on 21 October, says it has "not yet determined whether customer data was exfiltrated". Users who don't sign in through single sign-on have to reset their passwords, and integrations need their API credentials rotating. Then, on 22 October, its status page told customers to expect its BBO product to stay offline until the end of the week while it applies fixes for this week's critical PHP flaws. Two unrelated problems at one supplier, in one week.

Bullhorn also warned customers that criminals may pose as its staff and use the outage as cover to ask for passwords or codes. That is standard after any public supplier incident, and worth passing on to your team whenever one of yours goes down.

Three questions to answer this month

You cannot stop a generator failing in London. You can know in advance what it would do to you.

Which suppliers could you not work without for a day? Not every supplier, just the handful where a day's outage stops you invoicing, paying people, serving customers, or meeting a legal deadline. For most SMEs it is a short list. Write them down with an owner for each.

Where do they run, and what happens when that fails? Ask each critical supplier which cloud and region they use, whether they can carry on if one location fails, and when they last tested it. A good supplier will answer in plain English. "We use a leading cloud provider" is not an answer. Ask too whether their backup supplier runs on the same infrastructure as their main one.

What do you do on day one without them? For each critical supplier, agree the manual fallback before you need it. Who calls customers, how do you take payments, where is a recent export of your data, and how will you know it's back? Bullhorn offered customers data exports on request mid-incident. It is better to have your own before the outage.

Then check the contract for the parts that matter in a bad week: service credits, notice of incidents, and your right to your data in a usable format.

How Steelwise can help

Mapping which suppliers you actually depend on, where they run, and what you would do without them for a day is the kind of infrastructure review we do with clients. Get in touch.

Further reading

← All filings

We use cookies to understand how visitors use this site. More information.