Fluff Overflow

My heart goes out to the Blogger team — especially their support folks, who are dealing with the vitriol on Twitter; and their operations engineers, who have likely been working for 24+ hours straight to restore service.

Severe outages can happen anywhere, even at a Google-owned site that’s been around for 12 years and normally has absolutely phenomenal uptime. Last month’s AWS outage showed that sites hosted in the cloud aren’t immune either.

Large-scale web sites are enormously complex, with a nearly infinite number of ways that something can go wrong. And until the day when the metal ones come (…and they will…), these sites are still run by humans, and even the best engineering teams can make mistakes.

Software engineers try to build things to withstand hardware failures and human error alike, but no system is perfect.  When an outage happens, all we can do is restore, recover, rebuild, refactor, cry, swear, apologize profusely, and be thankful that we’re not doctors, air traffic controllers, or nuclear physicists.