Server Glitch Crashes T-Mobile Network

A software problem in NSN subscriber database servers shut down T-Mobile's network yesterday leaving 40M German customers without service

Michelle Donegan, Contributing Editor, Light Reading

April 22, 2009

2 Min Read
Server Glitch Crashes T-Mobile Network

T-Mobile Deutschland GmbH suffered a massive network outage Tuesday that left all of its 40 million customers unable to make calls or send texts for four hours.

The service collapse was caused by a software problem in the network that caused the Home Location Register (HLR) servers supplied by Nokia Networks to crash, which, with refreshing accountability, has issued an apology.

"We apologize to all T-Mobile customers for the service disruption," says a Nokia Siemens Networks (NSN) spokesman. "We're working closely with T-Mobile to identify the software problem and why the redundancy built into the system didn't kick in."

The HLR database holds all of a mobile operator's subscriber information -- such as location, preferences, account status -- which is critical for routing calls. If this database goes down, calls cannot be completed.

In T-Mobile's calamitous outage yesterday, the subscriber locations were not being updated on the central HLR database, according to an NSN spokesman. All the HLR servers had to be taken offline and brought back up one at a time, which explains why the outage lasted as long as four hours.

The HLR servers that went wrong yesterday were installed by T-Mobile as part of a major upgrade project to streamline its subscriber databases. The servers in question are part of the product line Nokia Siemens acquired when it bought Apertio Ltd. for $206 million in January 2008. (See T-Mobile Picks NSN, Nokia Siemens Snaps Up IMS Vendor, NSN Completes Apertio Buy, and LR Names 2008 Leading Lights Winners.)

Counting the costs
A T-Mobile spokeswoman would not comment on how much yesterday's network outage would cost the operator in lost revenues.

But Germany-based telecom executive Chris Larmour, chief marketing officer at Actix Ltd. , which provides automated network status management systems to operators such as Vodafone Group plc (NYSE: VOD), estimates it could be about $100 million.

Larmour believes the severity of T-Mobile's network outage yesterday could have been limited.

"Some of this could have been avoided," says Larmour (who, incidentally, is not a T-Mobile customer). "It went wrong and no one was able to manage it. It took them four hours to figure it out. It will take them months to get back to normal."

The outage could be costly, not just for T-Mobile in terms of lost revenues, damaged market perception, and service reputation, but also for NSN if T-Mobile decides to seek compensation.

After a similar network outage back in 2004 at Bouygues Telecom in France, the operator sued its HLR supplier, Tekelec , for $81 million in damages. (See Bouygues Sues Tekelec Over Outage.)

— Michelle Donegan, European Editor, Unstrung

About the Author(s)

Michelle Donegan

Contributing Editor, Light Reading

Michelle Donegan is an independent technology writer who has covered the communications industry on both sides of the Pond for the past twenty years.

Her career began in Chicago in 1993 when Telephony magazine launched an international title, aptly named Global Telephony. Since then, she has upped sticks (as they say) to the UK and has written for various publications, including Communications Week International, Total Telecom, Light Reading, Telecom Titans and more.

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like