Amazon Web Services Outage Caused by Typo

Mitch Wagner
3/3/2017

A typographical error re-configuring Amazon Web Services led to an outage that took down millions of Internet sites for hours this week.

The Amazon Web Services Inc. (AWS) Simple Storage Service (S3) team was debugging a problem that caused S3 billing to run more slowly than expected. An S3 team member executed a command at 9:37 am PT intended to remove a small number of servers for one of the S3 subsystems used for billing, according to a statement Thursday by Amazon. The team member typed incorrectly, and removed a larger number of servers than intended. Outages in the cloud service continued until 1:54 pm PT, but full recovery took longer as some services had to go through a backlog of work, Amazon says.

Affected sites included Netflix, Airbnb, Slack, and Light Reading.

Amazon apologized for the outage, and says it is making changes to its procedures to prevent recurrence. Among these: Changing the AWS Service Health Dashboard to reduce dependency on Amazon S3. Amazon had difficulty communicating the status of the outage to the public, because the status board depended on the problematic S3 service.

The outages cost $150 million to S&P 500 companies and $160 million to US financial services companies using the affected S3 infrastructure, according to Cyence, a firm that works with insurance companies to estimate cyber risk.

Related posts:

(2)  | 
Comment  | 
Print  | 
Related Stories
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
danielcawrey
danielcawrey
3/4/2017 | 4:01:06 PM
Re: Priced out
It's amazing to think how one small thing can cause such a huge problem. You would think it would be easier to just roll back services, but maybe it was hard to identify the problem? Either way, there's going to be a lot of IT meetings in the next few weeks on how to decrease Amazon dependency...
mendyk
mendyk
3/3/2017 | 9:52:20 AM
Priced out
I wonder how the $150 million to $200 million "loss" is determined. Also, do you think affected businesses will get any cost relief from AWS?
Featured Video
Upcoming Live Events
March 16-18, 2020, Embassy Suites, Denver, Colorado
April 20, 2020, Las Vegas Convention Center
May 18-20, 2020, Irving Convention Center, Dallas, TX
May 18, 2020, Hackberry Creek Country Club, Irving, Texas
September 15-16, 2020, The Westin Westminster, Denver
All Upcoming Live Events
Upcoming Webinars
Webinar Archive