O2 Boots Out Ericsson Server After Outages

After suffering two major service outages in the U.K., Telefónica UK Ltd. 's O2 said it will remove the central user database supplied by Ericsson AB (Nasdaq: ERIC) that took its network down on both occasions. (See Outage Strikes O2 UK.)

The rip-and-replace project will cost the operator £10 million (US$16 million), which is in addition to the £1.5 million ($2.4 million) that O2 claims it spends on its network each day.

In a statement issued on Wednesday, O2 COO Derek McManus said: "We are removing the Central User Database provided by one of our suppliers, which has suffered two different faults in the last few months. We are not prepared to risk this happening to our customers for a third time and are implementing a proven alternative solution."

An O2 spokesman said the operator had not yet decided which vendor or product would replace the Ericsson database.

On July 12, the O2 network was down for almost 24 hours in some places and left about 7 million customers without services. On Oct. 12, the operator suffered a similar outage that it says left 10 percent of its customers without service. The operator has 22 million customers in the U.K.

In response to O2's decision, Ericsson issued its own statement today:

    With reference to the O2 UK network disruption on Friday October 12, 2012 Ericsson can confirm that we, as a key supplier, worked closely with our customer to identify any contributing factors and immediately took necessary actions. The fault was fixed on the same afternoon. The issue was identified to be related to how the equipment was configured. O2 are moving to classic HLR which Ericsson has provided for many years in the network and continues to do so. We continue to work closely with O2 to ensure that service integrity is maintained.

Why this matters
Clearly, this is bad news for Ericsson. To be the supplier of the piece of equipment responsible for bringing down a customer's network and then being kicked out of that network is a blow.

O2's response today also reveals the cost to the operator of major network outages -- not just for equipment replacement, but also all the other investment they will need to make to restore confidence among their subscribers. It would seem that £10 million is only a start for O2.

For more
— Michelle Donegan, European Editor, Light Reading Mobile

Michelle Donegan 12/5/2012 | 5:18:58 PM
re: O2 Boots Out Ericsson Server After Outages

I updated the story with a statement from Ericsson.

Soupafly 12/5/2012 | 5:18:57 PM
re: O2 Boots Out Ericsson Server After Outages

This is indeed an embarassing and serious blow to Ericsson. You can bet your a$$ that AlcaLu, NSN, Huawei, ZTE, etc will all be using this as a PR weapon to beat them with.

The shambles in NZ with Alu RAN platform (or a "faulty" server as attributed) was a PR disaster for their RAN team & caused goodwill & reputational damage. T

The fact that this happened twice in a short space of time in a Tier1 operator/account with a piece of equipment that is critical to 3G/LTE networks rubs salt into the wounds.

If I was a betting man, I would hazard that this was a combination of;

a) Redback Networks. The new server smacks of their tech.

b) A software issue/ bug.

The Ericsson statement, as it stands, is just PR spun nonsense.

It fails to define (in anyway whatsoever) what might have occured.

If I was taking a wild guess I would suggest a replication bug that spread across the active/active nodes and lead to a DB corruption. The ripple was not detected quickly enough or propgated quicker than expected and the failback didn't happen, as originally modelled.

When your dealing with an active network with millions of active users & connections, any fail scenario that involves live data and has a revenue authentication and assurance function is the stuff of nightmares! Believe me!


yike 12/5/2012 | 5:18:52 PM
re: O2 Boots Out Ericsson Server After Outages

I would like to have the list of the happended issues for vendors to beat each other ... will that be helpful ?


<li>Orange France , July 2012: A major Friday evening service outage that affected about 26 million customers followed a software upgrade to Alcatel-Lucent (NYSE: ALU) HLR software carried out by staff from the vendor and the carrier. The upgrade introduced data inconsistencies that led to a rising tide of inconsistent messages being shared by Orange France's subscriber systems, which eventually became overloaded and unable to process new requests. The operator provided details of why and how the outage happened in testimony to the country's National Assembly: France takes these issues seriously. </li>
<li>Telenor ASA (Nasdaq: TELN), June 2011: An upgrade to a mobile broadband-related server resulted in the "most extensive breakdown that Telenor has experienced since the mobile network was established in 1993." Signaling traffic between various servers escalated and then "created disturbances" in other network elements, affecting the availability of voice and text message services to 3 million users. (See Telenor Explains Mobile Outage.) </li>
<li>Verizon Wireless , April 2011: The U.S. operator suffered problems with its 4G/LTE service that were linked to a Nokia Siemens Networks HSS. Verizon has subsequently suffered a number of interruptions to its 4G services, but the problems appear to have been caused by a different network issue each time. (See Verizon: LTE Is Back and Euronews: April 21.) </li>
<li>T-Mobile Deutschland GmbH , April 2009: The German giant's 40 million mobile customers were cut off for about four hours following a HLR system crash. The vendor supplier, NSN, held up its hands and issued an apology, a move not only to be applauded but one that would have helped T-Mobile get to grips with the fallout. (See Server Glitch Crashes T-Mobile Network.)

The good news is that there isn't a major danger of a service outage every time subscriber database software needs to be updated. "Regular HLR software upgrades happen all the time and they shouldn't be the cause of any problems," says Heavy Reading Senior Analyst Jim Hodges. "The major issues tend to come when there is a hardware upgrade or an operator is moving from a HLR to a HSS, or sometimes when there are a lot of new features in a software upgrade."

Those sorts of major upgrades are likely to become more commonplace as operators upgrade their packet core and subscriber data systems capabilities to support LTE services, so it's a safe bet that further reports of major mobile service outages will reach us during the next few years.


Sign In