& cplSiteName &

O2 Boots Out Ericsson Server After Outages

Michelle Donegan
LR Mobile News Analysis
Michelle Donegan

After suffering two major service outages in the U.K., Telefónica UK Ltd. 's O2 said it will remove the central user database supplied by Ericsson AB (Nasdaq: ERIC) that took its network down on both occasions. (See Outage Strikes O2 UK.)

The rip-and-replace project will cost the operator £10 million (US$16 million), which is in addition to the £1.5 million ($2.4 million) that O2 claims it spends on its network each day.

In a statement issued on Wednesday, O2 COO Derek McManus said: "We are removing the Central User Database provided by one of our suppliers, which has suffered two different faults in the last few months. We are not prepared to risk this happening to our customers for a third time and are implementing a proven alternative solution."

An O2 spokesman said the operator had not yet decided which vendor or product would replace the Ericsson database.

On July 12, the O2 network was down for almost 24 hours in some places and left about 7 million customers without services. On Oct. 12, the operator suffered a similar outage that it says left 10 percent of its customers without service. The operator has 22 million customers in the U.K.

In response to O2's decision, Ericsson issued its own statement today:

    With reference to the O2 UK network disruption on Friday October 12, 2012 Ericsson can confirm that we, as a key supplier, worked closely with our customer to identify any contributing factors and immediately took necessary actions. The fault was fixed on the same afternoon. The issue was identified to be related to how the equipment was configured. O2 are moving to classic HLR which Ericsson has provided for many years in the network and continues to do so. We continue to work closely with O2 to ensure that service integrity is maintained.

Why this matters
Clearly, this is bad news for Ericsson. To be the supplier of the piece of equipment responsible for bringing down a customer's network and then being kicked out of that network is a blow.

O2's response today also reveals the cost to the operator of major network outages -- not just for equipment replacement, but also all the other investment they will need to make to restore confidence among their subscribers. It would seem that £10 million is only a start for O2.

For more
— Michelle Donegan, European Editor, Light Reading Mobile

(3)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Michelle Donegan
Michelle Donegan,
User Rank: Light Beer
12/5/2012 | 5:18:58 PM
re: O2 Boots Out Ericsson Server After Outages

I updated the story with a statement from Ericsson.

User Rank: Light Sabre
12/5/2012 | 5:18:57 PM
re: O2 Boots Out Ericsson Server After Outages

This is indeed an embarassing and serious blow to Ericsson. You can bet your a$$ that AlcaLu, NSN, Huawei, ZTE, etc will all be using this as a PR weapon to beat them with.

The shambles in NZ with Alu RAN platform (or a "faulty" server as attributed) was a PR disaster for their RAN team & caused goodwill & reputational damage. T

The fact that this happened twice in a short space of time in a Tier1 operator/account with a piece of equipment that is critical to 3G/LTE networks rubs salt into the wounds.

If I was a betting man, I would hazard that this was a combination of;

a) Redback Networks. The new server smacks of their tech.

b) A software issue/ bug.

The Ericsson statement, as it stands, is just PR spun nonsense.

It fails to define (in anyway whatsoever) what might have occured.

If I was taking a wild guess I would suggest a replication bug that spread across the active/active nodes and lead to a DB corruption. The ripple was not detected quickly enough or propgated quicker than expected and the failback didn't happen, as originally modelled.

When your dealing with an active network with millions of active users & connections, any fail scenario that involves live data and has a revenue authentication and assurance function is the stuff of nightmares! Believe me!


User Rank: Light Beer
12/5/2012 | 5:18:52 PM
re: O2 Boots Out Ericsson Server After Outages

I would like to have the list of the happended issues for vendors to beat each other ... will that be helpful ?


<li>Orange France , July 2012: A major Friday evening service outage that affected about 26 million customers followed a software upgrade to Alcatel-Lucent (NYSE: ALU) HLR software carried out by staff from the vendor and the carrier. The upgrade introduced data inconsistencies that led to a rising tide of inconsistent messages being shared by Orange France's subscriber systems, which eventually became overloaded and unable to process new requests. The operator provided details of why and how the outage happened in testimony to the country's National Assembly: France takes these issues seriously. </li>
<li>Telenor ASA (Nasdaq: TELN), June 2011: An upgrade to a mobile broadband-related server resulted in the "most extensive breakdown that Telenor has experienced since the mobile network was established in 1993." Signaling traffic between various servers escalated and then "created disturbances" in other network elements, affecting the availability of voice and text message services to 3 million users. (See Telenor Explains Mobile Outage.) </li>
<li>Verizon Wireless , April 2011: The U.S. operator suffered problems with its 4G/LTE service that were linked to a Nokia Siemens Networks HSS. Verizon has subsequently suffered a number of interruptions to its 4G services, but the problems appear to have been caused by a different network issue each time. (See Verizon: LTE Is Back and Euronews: April 21.) </li>
<li>T-Mobile Deutschland GmbH , April 2009: The German giant's 40 million mobile customers were cut off for about four hours following a HLR system crash. The vendor supplier, NSN, held up its hands and issued an apology, a move not only to be applauded but one that would have helped T-Mobile get to grips with the fallout. (See Server Glitch Crashes T-Mobile Network.)

The good news is that there isn't a major danger of a service outage every time subscriber database software needs to be updated. "Regular HLR software upgrades happen all the time and they shouldn't be the cause of any problems," says Heavy Reading Senior Analyst Jim Hodges. "The major issues tend to come when there is a hardware upgrade or an operator is moving from a HLR to a HSS, or sometimes when there are a lot of new features in a software upgrade."

Those sorts of major upgrades are likely to become more commonplace as operators upgrade their packet core and subscriber data systems capabilities to support LTE services, so it's a safe bet that further reports of major mobile service outages will reach us during the next few years.


Featured Video
From The Founder
Light Reading founder Steve Saunders grills Cisco's Roland Acra on how he's bringing automation to life inside the data center.
Flash Poll
Upcoming Live Events
March 20-22, 2018, Denver Marriott Tech Center
March 22, 2018, Denver, Colorado | Denver Marriott Tech Center
March 28, 2018, Kansas City Convention Center
April 4, 2018, The Westin Dallas Downtown, Dallas
April 9, 2018, Las Vegas Convention Center
May 14-16, 2018, Austin Convention Center
May 14, 2018, Brazos Hall, Austin, Texas
September 24-26, 2018, Westin Westminster, Denver
October 9, 2018, The Westin Times Square, New York
October 23, 2018, Georgia World Congress Centre, Atlanta, GA
November 8, 2018, The Montcalm by Marble Arch, London
November 15, 2018, The Westin Times Square, New York
December 4-6, 2018, Lisbon, Portugal
All Upcoming Live Events
Hot Topics
Has Europe Switched to a Fiber Diet? Not Yet...
Ray Le Maistre, Editor-in-Chief, 2/15/2018
Net Neutrality: States' Rights vs. the FCC
Mari Silbey, Senior Editor, Cable/Video, 2/13/2018
IBM, Microsoft Duke It Out Over Chief Diversity Hire
Sarah Thomas, Director, Women in Comms, 2/15/2018
Will China React to Latest US Huawei, ZTE Slapdown?
Ray Le Maistre, Editor-in-Chief, 2/16/2018
5G: The Density Question
Dan Jones, Mobile Editor, 2/15/2018
Animals with Phones
Live Digital Audio

A CSP's digital transformation involves so much more than technology. Crucial – and often most challenging – is the cultural transformation that goes along with it. As Sigma's Chief Technology Officer, Catherine Michel has extensive experience with technology as she leads the company's entire product portfolio and strategy. But she's also no stranger to merging technology and culture, having taken a company — Tribold — from inception to acquisition (by Sigma in 2013), and she continues to advise service providers on how to drive their own transformations. This impressive female leader and vocal advocate for other women in the industry will join Women in Comms for a live radio show to discuss all things digital transformation, including the cultural transformation that goes along with it.

Like Us on Facebook
Twitter Feed