Operational issue – Multiple services (UAE)

This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…

Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.

A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.

“Oh, that’s in case there’s a fire!”

“What does it do?”

“It cuts… the power… uhh… for the safety of the fire department.”

“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”

“… yes.”

“Not 100% then, is it?”

> we will ensure we restore power and connectivity safely

this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost.

IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.

The more interesting question, is how many datacenters are just plonked next to a high-value military target?

For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases

> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.

Previous outage news makes it sound like the cloud providers still have quite a few logical single points of failure.

Exactly. 2 is only sufficient for HA against random failures. It's not enough for HA against a determined adversary willing to use targeted force.

That's so interesting. Are any of the US military (or other satellite state of the US) systems running in "normal" datacenters or do they have a few protected DoD datacenters in the US?

> Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.

The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few datacenter providers.

The larger threat to me with the lose of data is firstly the panic within public fronting services but also, with Hedge Funds, Pension funds or banking datacenters who might be using these and if they lose the data, then its gonna cause even more public mayhem.

Some might be saying oh off-site backups exist but there has atleast been one instance, where a single Google accident had led to massive issues for a 135 Billion $ pension fund.

Relevant Kevin Faang video about it: https://www.youtube.com/watch?v=3GOAUyipnM4 [Google Accidentally Deletes $135 Billion Pension Fund, Chaos Ensues]

This is the data center version of https://xkcd.com/538/. Realistically if there is a hot war what you’re saying seems accurate.

IIUC part of the reason ballistic missiles have multiple warheads is that some of them detonate high up to knock out air defenses and other electronics allowing the rest to fall through to their targets. The last time we tried this experiment as a species was the starfish prime tests in 1962 which caused some electrical havoc in Hawaii. These days our systems are probably more delicate and sensitive? All that is to say, in a scenario where nukes are going off I'm not sure you'd even need to target any datacenters in particular.. they're probably all toast by default.

It depends on the service if things move gracefully or not. The incident explains it's only EC2 (and dependent services) in that AZ, so if they try to route traffic for services hosted on EC2 to that AZ it's not working (and customers running instances in that AZ have lost access).

The other ones are not impacted. They always like to tell you to pay for more than one instance in different AZs so if this happens you don't get impacted.

Looking at Google Maps, there's Al Dhafra Air Base a couple of miles to the datacenter's south, an oil refinery a bit to the east, ports to the north, and a military academy to the west.

That's what I'm trying to understand too. It's this a meteor,tree,etc? Or a human made object,and if so accidental or intentional one. Further risk assessment would be dependent on root cause.

I actually like the way they said it. I don't know if it's a different cultural tradition, but the cool steely-eyed fact-based conversation always really felt so much more inspiring:

    Conrad: I got three fuel cell lights, an AC bus light, a fuel cell disconnect, AC bus overload 1 and 2, Main Bus A and B out.

    Aaron: Flight, EECOM. Try SCE to Aux.

> At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event. EC2 Instance launches will continue to be impaired in the impacted AZ. We recommend that customers continue to retry any failed API requests. If immediate recovery of an affected resource (EC2 Instance, EBS Volume, RDS DB Instance, etc.) is required, we recommend restoring from your most recent backup, by launching replacement resources in one of the unaffected zones, or an alternate AWS Region. We will provide an update by 12:30 PM PST, or sooner if we have additional information to share.

This has that same mechanical tone of an ice-cold captain dealing with a proximate situation providing exactly the information they know. No flavour commentary. Amazing. I fucking love it.

Maybe a missile, maybe a drone, maybe debris

Doesn’t really matter, we know trumps latest war is the cause

Potential moon bear attack; we're waiting on satellite imagery to confirm it. https://youtu.be/pvjgIxuVdo4?t=96

A factory not working because of a missile strike seems pretty classic actually.

When I first learned that there were AWS Middle East regions, my first thought was "wow they are more optimistic than I am ."

Not really. It's just been pretty damn quiet for years.

[dead]

UAE. Abu Dhabi

Historically in the US, some portion of Bell installations were designed to be resistant to attack. But it comes at large expense for construction and maintenance. Underground facilities also bring increased risk of flooding.

Competition and deregulation and lack of attacks leads towards less robust installations to reduce costs. Geographically redundant installations help as long as all installations aren't targetted; and are valuable for operational concerns other than just attacks.

> will there be a demand (all be small) for nuclear bunker esque datacenters

Those already exist. See for example Bahnhof's "Pionen - White Mountain" data center in Stockholm, or Cyberfort's "The Bunker" a bit west of London.

> but given the situation, How does that prevent the situation from happening again

You don't. Instead, you make sure your failover or DR setup is regularly tested and works.

Data centers are usually built to withstand local natural risks e.g. weather. All bets, SLAs, and insurance are usually off when it comes to acts of war.