& cplSiteName &

Amazon Web Services Outage Caused by Typo

Mitch Wagner

A typographical error re-configuring Amazon Web Services led to an outage that took down millions of Internet sites for hours this week.

The Amazon Web Services Inc. (AWS) Simple Storage Service (S3) team was debugging a problem that caused S3 billing to run more slowly than expected. An S3 team member executed a command at 9:37 am PT intended to remove a small number of servers for one of the S3 subsystems used for billing, according to a statement Thursday by Amazon. The team member typed incorrectly, and removed a larger number of servers than intended. Outages in the cloud service continued until 1:54 pm PT, but full recovery took longer as some services had to go through a backlog of work, Amazon says.

Affected sites included Netflix, Airbnb, Slack, and Light Reading.

Amazon apologized for the outage, and says it is making changes to its procedures to prevent recurrence. Among these: Changing the AWS Service Health Dashboard to reduce dependency on Amazon S3. Amazon had difficulty communicating the status of the outage to the public, because the status board depended on the problematic S3 service.

The outages cost $150 million to S&P 500 companies and $160 million to US financial services companies using the affected S3 infrastructure, according to Cyence, a firm that works with insurance companies to estimate cyber risk.

Related posts:

(2)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Featured Video
From The Founder
Ngena's global 'network of networks' solves a problem that the telecom vendors promised us would never exist. That doesn't mean its new service isn't a really good idea.
Flash Poll
Upcoming Live Events
March 20-22, 2018, Denver Marriott Tech Center
March 22, 2018, Denver, Colorado | Denver Marriott Tech Center
March 28, 2018, Kansas City Convention Center
April 4, 2018, The Westin Dallas Downtown, Dallas
April 9, 2018, Las Vegas Convention Center
May 14-16, 2018, Austin Convention Center
May 14, 2018, Brazos Hall, Austin, Texas
September 24-26, 2018, Westin Westminster, Denver
October 9, 2018, The Westin Times Square, New York
October 23, 2018, Georgia World Congress Centre, Atlanta, GA
November 7-8, 2018, London, United Kingdom
November 8, 2018, The Montcalm by Marble Arch, London
November 15, 2018, The Westin Times Square, New York
December 4-6, 2018, Lisbon, Portugal
All Upcoming Live Events
Hot Topics
Dell CTO: Public Cloud Is 'Way More Expensive Than Buying From Us'
Mitch Wagner, Mitch Wagner, Editor, Enterprise Cloud, Light Reading, 3/19/2018
Eurobites: Cambridge Analytica Feels the Heat
Paul Rainford, Assistant Editor, Europe, 3/20/2018
Is Business Voice Rapidly Fading?
Carol Wilson, Editor-at-large, 3/15/2018
HR: Cable Dominates US Broadband
Carol Wilson, Editor-at-large, 3/21/2018
Animals with Phones
Live Digital Audio

A CSP's digital transformation involves so much more than technology. Crucial – and often most challenging – is the cultural transformation that goes along with it. As Sigma's Chief Technology Officer, Catherine Michel has extensive experience with technology as she leads the company's entire product portfolio and strategy. But she's also no stranger to merging technology and culture, having taken a company — Tribold — from inception to acquisition (by Sigma in 2013), and she continues to advise service providers on how to drive their own transformations. This impressive female leader and vocal advocate for other women in the industry will join Women in Comms for a live radio show to discuss all things digital transformation, including the cultural transformation that goes along with it.

Like Us on Facebook
Twitter Feed