Update: Software Load Balancers Blamed for Google Outage

Earlier today, GMail suffered widespread outages, preventing many from accessing their e-mail, and other Google Web Apps

Update: Earlier this week Gmail suffered widespread outages preventing many from accessing their email, gdocs, gdrive and other google based online applications. Below is a description as to how and why the network interruptions occurred:

Load Balancers Blamed

According to Google sources, a bug in a routine software update caused load balancing equipment to misread the status of a data center as “unavailable”. Thankfully, Google data centers are equipped with failsafes that prohibit the entire network from going down, which is the reason why only gmail, gdocs and gdrive services were affected. The bugged software update is indeed what prevented user requests from being processed and routed through google servers assigned to such tasks.

Browser Crashes Explained

Although considered to be a residual effect, the software bugs are to blame for the web browser crashes as well. As a result, users who rely on Google Sync – to sync multiple devices to personal browser settings experienced un-explained crashes. Google engineers explained that once the backend servers became overwhelmed with service request to sync, they crashed. In hopes of avoiding such service disruptions in the future, google says it is changing its protocol for rolling out new software updates to load balancer equipment. Given the complex layout of Load Balancers, special care will have to be taken to ensure future rollouts of updates are tested and contained before deployment into the broader network. Google says that the aforementioned process is currently under review.

Original Post

Earlier today, Gmail suffered widespread outages, preventing many from accessing their email, gdocs, gdrive and other google based online applications.

Although services were only down for 20 minutes, this outage is perhaps the oddest Google has experienced in the years since gmail’s inception. What’s going on at Google’s Data Center? Un-officially speaking, application bugs may be to blame.

The story of the outage spread quickly this morning through social media outlets Facebook and twitter. Around 12:30 pm ET, Google reported that they were indeed investigating reports that outages had occurred, yet did not admit that outages had taken place. Officially, Google has referred to this little hiccup; as a “service disruption. Nice save Google.

Error Codes Received

Call it what you will, Google’s little “service disruption” prompted many 502 server error messages to flash across screens nationwide. Although no official statements or explanations had been released via Google’s twitter page at the time of this writing, Gmail services for businesses were impacted. Various reports on twitter confirmed that Chrome browsers had crashed, preventing access to Google Calendar, Google drive and cloud based email services. This seems like a lot of trouble not to merit an official statement from google’s twitter page.

Given the popularity of Google’s services, and widespread usage amongst its customers, 20 minutes of downtime is somewhat of an anomaly. The last reported “service disruption” for gmail users was this past April, where over 35 million users were left without access to their gmail accounts, in some reports, nearly 36 hours passed before services were restored. At least with an outage that long, google was cool enough to send out a tweet. I guess they figured 20 minutes of downtime is nothing to write home about. If you or your company was affected by the outage this morning, we sincerely hope business has continued as per usual this afternoon. We’ll be sure to tweet any updates on our official twitter page.