A Cloud Reboot Causes Panic At Netflix

Raise your hand if you’ve ever watched an entire season of a favorite television series in one sitting, realized it’s the morning of the next day, called in sick, and then watched the second season all because, well, you could?

*Looks around the room* Everyone’s hand was raised, no surprise there. And if I were to poll you on if you’ve done that multiple times, I’m sure those hands would stay up, right?

In fact, some people I know exclusively use Netflix for their source of television and don’t subscribe to cable or satellite providers at all.

How devastating, then, would it be for Netflix to be completely unavailable due to that mysterious villain known as “The Cloud?” I would say nationwide panic would ensue—more than likely of biblical proportions whenever the Bible is revived from the desecration of humanity no-Netflix would cause.

Well, good news! That nearly happened! Oh, wait. That’s bad news.

What Happened To Netflix’s Cloud?

During the great server reboot of 2014 (one to apply a security patch, no less) several major cloud providers were affected including Amazon, Rackspace, and of course, Netflix.

Get ready for some Netflix vernacular, coming at you. Hard.

While you were on the fourth viewing of the fifth season of The Office, Netflix was sipping back coffee from golden mugs watching Bridezillas on one diamond-plated monitor, while watching their bank account go higher and higher on another. Of course, they knew their new security patch (Xen Security—which just sounds sneaky) was installing and they were in the process of rebooting their Cassandra database which has over 2,700 production Cassandra Nodes but only 218 were set to be rebooted (vernacular, man).

Well, 22 of those failed to reboot normally and Netflix put down their gold cups and sweat beaded down their hairy-but-exfoliated back.

What Is Netflix Doing to Ensure This Never Happens Again?

Luckily for Netflix, they have an automated program which detected all of this and replaced them without Netflix having to do much work with the humans they control (automated data centers, bro).

Even more luckily for Netflix, they had tested their servers with a program called “Chaos Monkey” which tested the resiliency of their servers just a few months back.

Here’s a quote from Christos Kalantzis, Netflix’s manager for cloud database engineering and two-time best name at Netflix award winner:

“When we got the news about the emergency EC2 reboots, our jaws dropped. When we got the list of how many Cassandra nodes would be affected, I felt ill. Then I remembered all the Chaos Monkey exercises we’ve gone through. My reaction was, “Bring it on!”

Now how come all these “data scares” seem to have their genesis at “The Cloud?” I mean, there was “Celebgate” just a few weeks back and other various instances of the cloud basically sucking.

Come on, Netflix. Throw as many Chaos Monkeys, Beavers, Toucans, or whatever you want at your servers. Nothing beats the security of staying grounded with some nice, reliable, secure dedicated servers.

I’m sure your customers wouldn’t even know the difference and they can continue to watch whatever was recommended to them under the “Sand and Sandals” genre and you can keep taking all your money to the bank (while taking off more beloved shows, but that’s a blog for another day).