AWS and friends put genAI to work on network troubleshooting

Every telco has been there: The network engineer reports a 99.9% call connection rate while the customer experience team complains that dropped calls affected the most valuable customers. Cue a lengthy process that involves crunching hundreds of terabytes of data across numerous back-end systems to figure out the problem. Hours or even days later, the answer emerges.

"Vodafone or another large telco could be generating one petabyte per day in the OSS [operational and support systems] domain," said Fawad Qureshi, the chief technology officer for the global industry field at Snowflake, a data-cloud company. "It is a needle-in-a-haystack problem."

AWS and friends, including Snowflake, now think they have a speedier approach. It's thanks to the magic of generative AI, and it's being shown off this week in Copenhagen – along with numerous other genAI goodies – where the telecom industry's chief boffins are gathered for the TM Forum's annual shindig. By pooling data from multiple sources and running it through a large language model, the companies say they can furnish perplexed engineers with quick responses to technical queries. The hours and days spent trying to resolve a problem could feasibly shrink to minutes.

GenAI is set to be a main theme of Digital Transformation World this week. (Source: Pitinan Piyavatin/Alamy Stock Photo)

The fast-growing Snowflake – whose revenues climbed 37% year-on-year for the second quarter, to $640 million – claimed to process about 2.6 billion data queries in the cloud for its various customers and partners in the month of January alone. Its job here is to compile data on cell towers, physical addresses and points of interest, and combine this with customer loyalty information and details harvested from billing support systems (BSS) and OSS.

Snowflake is not the only company involved in the data-gathering and analysis process. The so-called performance management data collected from OSS is in a complicated format known as ASN.1. Here, the system relies on DigitalRoute, another data-management company with telco-specific expertise. DigitalRoute's main job is to capture those ASN.1 files, decipher them and send the resulting data to Snowflake. Billing data is also fed into Snowflake from DigitalRoute.

Does genAI dream of electric Llamas?

Eventually, the companies end up with something they can hand over to genAI. An AWS platform called SageMaker can apparently host numerous large language models that would formulate queries and provide relevant responses. The demo shown in Copenhagen uses Llama 2, an open-source foundation model fathered by Meta, but SageMaker has also been tested with Claude, the product of an AI startup called Anthropic, which had reportedly raised as much as $1.5 billion in funding by July.

Reliance on large language models also used by homework-shy students and lazy reporters will inevitably beg questions about their suitability for telecom. Llama 2, Claude and others have not been trained specifically on telco data, but Qureshi says they can be appropriately educated. "These models are more general purpose. They are not telco models," he said. "There is a process of fine tuning and adapting parameters."

A related concern is whether confidential company data could be used to finetune a model that subsequently aids a competitor. What's more, in the example demonstrated by AWS, DigitalRoute and Snowflake, the model is hosted inside one of the Internet giant's public cloud facilities, vast repositories of data where systems handling multiple companies are stacked like books at a library. Amid some high-profile complaints about copyright infringement by large language models, all this could twang a few nerves about data privacy.

However, the models used by AWS and Snowflake can be finetuned not just for an individual client but also for each use case of that client, according to Qureshi. Systems can then be roped off from public consumption, largely by ensuring that a telco's existing security setup remains a part of the solution.

"You cordon off the large language model and no data leaves the customer environment," said Qureshi. "This is a private large language model running in your domain bounded by security parameters." The concept of bringing the application to the data rather than the data to the application is one that AWS and Snowflake are emphasizing in the literature about this technology. Besides guarding against any data leakage, this would ensure huge volumes of data do not have to be moved around.

Perhaps the main worry for any telco engineer is the phenomenon of hallucinations, whereby genAI invents information, messes up facts and even goes entirely rogue. In one of the most famous examples, a New York Times reporter was told to leave his wife by ChatGPT, which later urged him to call it Sydney. For Qureshi, all that helpfully reinforces his point that genAI is not a replacement for people.

"The word we use is copilot," he said. "Would you allow one to land the plane? We don't think we should be running large language models on autopilot. We are using them to improve productivity in an organization, not to replace humans."

Job fears

That probably won't stop telco-sector employees from wondering about job losses to genAI. The workforce across some of the world's biggest operators has shrunk by hundreds of thousands in recent years, with automation and "digitalization" held partly responsible for the cuts. Some company bosses have also said they expect to cut thousands of jobs thanks to investment in AI. In May, current BT CEO Philip Jansen told reporters that he reckoned about 10,000 jobs at the UK telco – which employs roughly 130,000 people in total – would be replaced by AI this decade.

The use case demonstrated in Copenhagen sounds incredibly niche. Qureshi said it was chosen because that specific issue has affected nearly all telcos. But the same underlying platforms and software technologies could support a much broader range of network and operational needs. Levels of interest in Copenhagen this week could be a useful gauge of how far this might go.