AT&T's outage twists up its MWC story

AT&T officials were headed to the MWC Barcelona trade show with plenty to trumpet: A historic agreement with Ericsson, a new embrace of open RAN, and a cutting-edge core network shift into Microsoft's Azure hyperscale environment.

And then the outage happened.

On Thursday morning at around 4 a.m. ET reports began to emerge that AT&T's network had collapsed in markets including Dallas, Houston, Los Angeles and Atlanta. According to Ziff Davis' Down Detector website, at least 75,000 customers were affected.

To be clear, network outages, including on wireless networks, are relatively common. Indeed, Optus in Australia said its recent network outage cost the company $40 million.

But the scale of AT&T's outage – and presumably a relative lack of other newsworthy items on a Thursday morning – sent US media outlets both big and small into overdrive:

"Some of our customers are experiencing wireless service interruptions this morning," AT&T said in a statement to a number of outlets. The operator said it is "working urgently to restore service to them." It also recommended that impacted AT&T subscribers use "Wi-Fi calling until service is restored."

Related:AT&T network outage frustrates FirstNet users

AT&T's rivals took a victory lap:

"Verizon's network remains fully operational. Some customers may have experienced issues this morning when calling or texting those served by another carrier. Our network continues to function normally," the operator said in a release Thursday morning.

"Decentralized wireless (DeWi) helps mitigate these type of disruptions. @REALLYWIRELESS, built on our REALLY DeWi network has not been impacted by these outages," Really Wireless CEO Adam Lyons posted to social media.

T-Mobile officials too have said the company's operations haven't been affected.

So what's going on?

Informed speculation

AT&T officials have not yet responded to questions from Light Reading about the cause of the outage. That's not a surprise considering the company is in the midst of triage. In an updated statement to Cnet later Thursday morning, the company said that service had been restored to 75% of its network.

UPDATE: On Friday, AT&T said the outage was over and that it was "caused by the application and execution of an incorrect process used as we were expanding our network, not a cyber attack. We are continuing our assessment of today’s outage to ensure we keep delivering the service that our customers deserve."

Regardless, the situation could involve a software upgrade to AT&T's core network. That upgrade could have affected AT&T's wired transport operations – the fiber connections used by AT&T and other companies to shuttle Internet data from one part of the county to another – as well as AT&T's wireless connections. It's also possible that AT&T's core network upgrade is somehow tied to the settings inside its customers' phones, based on reports that some customers were affected and that others – including those using the same cell site – were not.

"Typically when something like that happens, it's something in the core," analyst Earl Lum, of EJL Wireless Research, told Light Reading on Thursday.

Lum explained that problems with core network upgrades can often cascade across the country.

"This is the downside to automation," agreed analyst Roger Entner of Recon Analytics. He told Light Reading on Thursday that operators including AT&T are working to shift their operations from legacy hardware solutions to software-based products that are cheaper to operate. "That's the beautiful thing about automation."

However, that shift also allows glitches and bugs to quickly spread far beyond a particular radio or cell site.

Entner pointed out that the time of the outage - in the early morning - is typically when software updates are deployed.

AT&T's core story

A glitch inside AT&T's core network would come at a time when the operator is making major changes to its core operations. The operator is in the midst of shifting traffic to its new standalone (SA) 5G core via its 2021 core network agreement with Microsoft.

Under AT&T's agreement with Microsoft, the hyperscale cloud company is taking AT&T's Network Cloud technology, building it into Microsoft's standard hybrid cloud product, and then delivering that carrier-grade hybrid cloud solution back to the market and AT&T itself. AT&T has said it will run those core operations in different "edge zones" around the country.

As for AT&T's shift to the SA version of 5G, AT&T's chief networking executive, Chris Sambar, said recently that the operator is now moving some customers to standalone 5G. "Many of the newest mobile devices are ready for 5G standalone, and we continue to move thousands of customers every day," he wrote. That represents a significant delay from AT&T's original plans to launch standalone 5G in 2020.

To be clear, the industrywide shift to standalone 5G has posed challenges to a number of operators. That's because installing the technology requires changes not only to the core brains of an operator's network, but also the distribution and management of phones capable of connecting to that SA core.

Thus, it's possible that AT&T's outage involves the operator's shift to an SA 5G core because it appears to involve settings in the network as well as on customers' phones.

Regardless: "At a high level, we believe this helps support our view that AT&T's 5G network has fallen behind peers," wrote the financial analysts at KeyBanc Capital Markets in a note to investors Thursday.

That's probably not what AT&T officials had hoped to hear as they make the trek to Barcelona, Spain, for the annual MWC trade show.