Dangerous things, software updates. Push the wrong button and they can be as devastating as any Chinese cyberattack, plunging parts of the Internet into darkness. The lights temporarily went out for some of the world's most popular websites last month when a customer of Fastly, a content delivery network (CDN), stumbled over a nasty bug in some code. This month's culprit is Akamai, another CDN whose keyboard-tapping engineers ran into a similarly pestilent bug and brought down websites including Barclays, HSBC, British Airways and Airbnb, as reported by Sky News.
Brief as the outages were – both were fixed within an hour – they are a stomach-churning reminder of our growing dependence on Internet technology provided by a relatively small number of companies. Allowing critical systems to ride together in the same bug-infested vehicle seems as risky as cramming world leaders into a single sputtering Tupolev jet (although some might welcome the latter).
Akamai Summarizes Service Disruption (RESOLVED)— Akamai Technologies (@Akamai) July 22, 2021
At 15:46 UTC today, a software configuration update triggered a bug in the DNS system, the system that directs browsers to websites. This caused a disruption impacting availability of some customer websites. (1/3)
Apologists for Akamai and Fastly will point out that bugs were dealt with quickly, minimizing the economic impact. But there is no guarantee that problems will always be so easy to remedy, especially if they arise externally. Akamai acknowledges in its own recent quarterly filing with the US Securities and Exchange Commission that: "Cybersecurity breaches and attacks on us … could lead to significant costs and disruptions that would harm our business, financial results and reputation."
It neglects to mention the ramifications for its customers, and any cyber villain will have noted this week's outage and the precise ripple effect that an attack on Akamai would trigger. Hackers will regard the CDN as a point where they can inflict maximum damage, much as a martial artist looks for body parts where a strike would be disabling.
The Internet Society, a US not-for-profit group, drew attention to the risks of overreliance on a few Internet companies in a 2019 report called "Consolidation in the Internet economy." Akamai is named in it as one of just four dominant providers in the market for DNS (domain name server) hosting, alongside AWS, Cloudflare and Dyn (now a part of Oracle). By May 2017, those companies had a combined market share of 50%, the report authors reckoned, as organizations abandoned "self-hosting" and turned to the cloud.
Meanwhile, some 87.5% of the world's top 1,000 websites were relying on CDNs by August 2018, up from only 50% in June 2014. Of the websites the Internet Society examined, around 27% were using Amazon Cloudfront and another 27% were on Akamai. "The fact that 474 of the top 1,000 global websites use one of these two CDN providers indicates that they have significant market share," said the report.
ThousandEyes, a network intelligence company owned by Cisco, noted the full impact of the Akamai blackout in a statement that shed light on the incident. "Though brief, the scope of the outage impacted many sites and applications ranging from gaming sites to major banks, airlines and more that leverage the Akamai CDN service," said the firm. "DNS is a critical first step in reaching a web property and while Akamai's CDN service does not appear to have been impacted, it was unreachable for many users during the incident."
Investors in Akamai seemed unfazed by the short outage yesterday, with the company's share price down just 0.56% when markets closed. Like other cloud companies, Akamai has been on a roll, its quarterly revenues hitting about $843 million earlier this year, up from $568 million in the same part of 2016. Net income has grown from $75 million to $156 million over that period. There must seem little risk that clients go back to self-hosting. And if bugs can upset a company generating $3.2 billion in annual sales – armed with 325,000 servers and employing around 8,400 people – they can probably hurt anyone in the CDN game.
These two recent outages coming so close together should be sobering for a telecom sector that has become increasingly reliant on the public cloud. Most of the big operators in Europe and North America now have some arrangement with an Internet giant, be it AWS, Google Cloud or Microsoft Azure. Some are even putting the nerve centers of their new 5G networks in the public cloud. Any telecom downtime caused by an Akamai-like incident could send shock waves throughout the economy.
- Fastly outage gives preview of Internet apocalypse
- Bell Canada is latest telco to succumb to AWS at the edge
- It's time for telecom to worry about the public cloud
- Dish fealty to AWS risks 'vendor lock-in' and open RAN fallout
- Singtel partners Azure as public clouds continue edge advance
— Iain Morris, International Editor, Light Reading