Cloud outages will happen
A bad cloud outage can take a large part of the internet with it for hours, as we’ve seen time and again. This year both Google Cloud and Azure were down for hours, one from a utility grid power failure, the other from a fibre-cut during maintenance. There will be more before the year is out. We can’t prevent them, but we can decide how much of our own product goes down with them.
Things you can do now
The usual weak spot is running everything in one region, on one database, with no copy of the data anywhere else. It’s the cheapest way to start, and it’s fine right up until the day that region has trouble, when there’s nothing to fail over to and you’re down for as long as the provider is. A few cheap habits change that:
Spread across availability zones. They’re separate data centres inside the same region, and running in two or three of them costs little but survives one of them failing.
Keep a copy of your data somewhere else, another region or another provider, and check now and then that you can restore it.
Write a short runbook: who does what when things break, and how you’ll tell customers. A status page beats silence.
Design to degrade. Showing a read-only or slightly stale version of your product is far better than a blank error page.
What to do after one
When an outage does hit, run a short, blameless post-mortem afterwards: what broke, and the one change that would help most next time. You don’t need to build for a once-a-year event as if it’s a daily one. You need to close the cheap gaps before the next one finds them.
