Google apologizes for outage, claims what we had was a failure to configurate...

Google apologizes for outage, claims what we had was a failure to configurate...

Google services like Gmail, Calendar, Google+, and pretty much anything and everything that required a login was down for a while today, and it turns out the reason for it all was a bum configuration file that got accidentally, devastatingly pushed live. Ben Treynor, VP Engineering at Google:

At 10:55 a.m. PST this morning, an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users' requests for their data to be ignored, and those services, in turn, generated errors. Users began seeing these errors on affected services at 11:02 a.m., and at that time our internal monitoring alerted Google's Site Reliability Team. Engineers were still debugging 12 minutes later when the same system, having automatically cleared the original error, generated a new correct configuration at 11:14 a.m. and began sending it; errors subsided rapidly starting at this time. By 11:30 a.m. the correct configuration was live everywhere and almost all users' service was restored.

Google apologized and said additional steps would be taken to try and prevent such things from happening again in the future. (And I apologize for making up words in the title, even if they're funny.)

Just another reminder that every service — every service — goes down from time to time, and it's how they handle it when they do that matters. Also a reminder of how dependent most of us are on online services these days. Seriously, what's worse: A power outage when you still have internet, or an internet outage when you still have power?

Have something to say about this story? Leave a comment! Need help with something else? Ask in our forums!

Rene Ritchie

EiC of iMore, EP of Mobile Nations, Apple analyst, co-host of Debug, Iterate, Vector, Review, and MacBreak Weekly podcasts. Cook, grappler, photon wrangler. Follow him on Twitter and Google+.

More Posts



← Previously

Hay day: Top 6 tips, tricks, and cheats to save cash and grow your farm fast!

Next up →

Deal of the Day: Amzer Soft Gel TPU Gloss Skin Case for iPhone 5/5S

Reader comments

Google apologizes for outage, claims what we had was a failure to configurate...


"If you were affected by the outage, let me know for how long, and whether or not it made you uncomfortable"

I'm always surprised by how many people are bothered by outages for a service that is free... you don't pay for. There really isn't much right to complain when you aren't paying a thing for what they are providing.

I pay for it. I have 2 Google Apps accounts, both paid.

Also, many people pay for "free" with their personal data, which can be more valuable than money.

Google has immense user debt, so outage frustration is absolutely understandable.

I am pretty sure you are in the minority here Rene, however if you do pay for this service then you have all the right in the world to be up in arms about this. I agree with the OP though since I do not pay for their services. They're a luxury for me and so if the servers are down once in a blue moon I can live with it.

Jokes aside, I wonder how expensive the outages really are for Google? I'm sure some math whiz on the web has a rough guess based on earnings statements?

Anything man-made is prone to failure. Be thankful for the quick transparency.

Posted via the Android iMore App!

Didn't really there was an outage here in my area. Then again, I'm not a big email user so an hour or two of outage is not really a big issue for me.

Didn't even notice and Google did an excellent job getting services restored within an hour. (Blackberry could learn a few pointers.)

We take for granted the amount of work it takes to maintain these "man-made" systems and for the "outage" to be down less hour is very good.

Well I'm glad to see they are back up and running. I don't use google services that much.

Sent from the iMore App

"...Ben Treynor, VP Engineering at Google:..."

And after the quote: "Taylor apologized and said additional steps would be taken to try and prevent such things from happening again in the future..."

So was his name Treynor or Taylor? Or are we talking two completely different people here?

Mine was down for what seemed like an hour. I use Gmail for my company email (with my corporate domain name being my @.x address), so it was definitely a disturbance. But I've had outages with other ISP's, so I wasn't that bothered. Google's relative uptime is decent in my experience. That being said I'm considering leaving Gmail altogether because of Google's recent attempts to forcefully require users to connect all of its services into one monopolistic ecosystem of data mining.