Keyboard error caused an estimated $150M+ outage this week.

Mitch Wagner, Executive Editor, Light Reading

March 3, 2017

1 Min Read
Amazon Web Services Outage Caused by Typo

A typographical error re-configuring Amazon Web Services led to an outage that took down millions of Internet sites for hours this week.

The Amazon Web Services Inc. (AWS) Simple Storage Service (S3) team was debugging a problem that caused S3 billing to run more slowly than expected. An S3 team member executed a command at 9:37 am PT intended to remove a small number of servers for one of the S3 subsystems used for billing, according to a statement Thursday by Amazon. The team member typed incorrectly, and removed a larger number of servers than intended. Outages in the cloud service continued until 1:54 pm PT, but full recovery took longer as some services had to go through a backlog of work, Amazon says.

Affected sites included Netflix, Airbnb, Slack, and Light Reading.

Amazon apologized for the outage, and says it is making changes to its procedures to prevent recurrence. Among these: Changing the AWS Service Health Dashboard to reduce dependency on Amazon S3. Amazon had difficulty communicating the status of the outage to the public, because the status board depended on the problematic S3 service.

The outages cost $150 million to S&P 500 companies and $160 million to US financial services companies using the affected S3 infrastructure, according to Cyence, a firm that works with insurance companies to estimate cyber risk.

Related posts:

About the Author(s)

Mitch Wagner

Executive Editor, Light Reading

San Diego-based Mitch Wagner is many things. As well as being "our guy" on the West Coast (of the US, not Scotland, or anywhere else with indifferent meteorological conditions), he's a husband (to his wife), dissatisfied Democrat, American (so he could be President some day), nonobservant Jew, and science fiction fan. Not necessarily in that order.

He's also one half of a special duo, along with Minnie, who is the co-habitor of the West Coast Bureau and Light Reading's primary chewer of sticks, though she is not the only one on the team who regularly munches on bark.

Wagner, whose previous positions include Editor-in-Chief at Internet Evolution and Executive Editor at InformationWeek, will be responsible for tracking and reporting on developments in Silicon Valley and other US West Coast hotspots of communications technology innovation.

Beats: Software-defined networking (SDN), network functions virtualization (NFV), IP networking, and colored foods (such as 'green rice').

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like