At the time of the outages on the "XT" 3G network in December 2009, and January and February this year, Telecom New Zealand CEO Paul Reynolds said that he had put his network supplier, Alcatel-Lucent, "on notice." (See New Zealand's 3G Network Nightmare.)
Now, following the publication of an independent review by Analysys Mason of the network's design, build, and operation at that time, Reynolds says that AlcaLu remains on notice. (See Telecom NZ Reports on 3G Outage and Telecom New Zealand Reports Q3.)
The independent network review, which was commissioned by Telecom New Zealand, does not actually blame or exonerate Alcatel-Lucent or Telecom New Zealand for the network failures, but Reynolds is still taking a tough stance with his supplier.
"As a supplier Alcatel has to earn its stripes," said Reynolds on the third-quarter earnings call with media and analysts on Friday. Reynolds noted that those were in fact the words Alcatel-Lucent CEO Ben Verwaayen recently used to describe the current situation with the operator and that he agreed with Verwaayen's conclusion.
While Reynolds wants his supplier to prove itself, he would not be drawn on what impact the outages will have on the operator's future with Alcatel-Lucent. "Our detailed relationship with Alcatel-Lucent is clearly commercially confidential," he said.
Speculation has already circulated, however, that Telecom and Alcatel-Lucent have already discussed a compensation deal that would require AlcaLu to credit Telecom with NZ$100 million (US$72 million) for the damages, according to a report in local paper, The Dominion Post, which cites unnamed sources and was picked up by Dow Jones Newswires. An Alcatel-Lucent spokesman tells Light Reading Mobile that the company does not comment on rumors or speculation.
What went wrong with XT?
The 3G network outages affected some 200,000 customers in New Zealand and cost the operator NZ$15 million ($10 million) in customer compensation payouts. The network calamity also resulted in the resignation of Telecom's CTO and the head of Alcatel-Lucent's New Zealand operation. (See Telecom New Zealand CTO Resigns, AlcaLu Replaces Kiwi Chief, and Telecom New Zealand Apologizes.)
Analysys Mason was tasked with analyzing the network, which is designed, built, and operated by Alcatel-Lucent, at the time of the outages. The consultancy's basic conclusion is that the network architecture aligned with industry best practices, but the network and supporting operations were "not ready" to handle large amounts of data traffic that came onto the network.
Specifically, the report lists five key findings:
- The network failed because the network and supporting operations were not ready to manage the levels of traffic it experienced;
- Software issues contributed to network instability;
- Although the XT network was designed to initially provide planned coverage that matched the CDMA network, the initial configuration of the XT network and some network build issues led to coverage variability;
- Some aspects of the network architecture are overly complex meaning that any faults are difficult to find and rectify; and
- Immature operational management systems and process failures contributed to the impact of network issues.
Regarding the "immature" operational management, Reynolds explained that even though the customer growth was within the operator's forecast, the customer growth plan needed to be more detailed.
"You not only need to get those numbers of customers and the number of calls... you need to know exactly where they are, the time-of-day calling profiles, and detail the splits between data, voice and SMS, because it's quite sensitive to very local variations," he said. "So the capacity management and the operational management has had to get into much more detail in that sense."
Reynolds said that the work needed to meet the recommendations that Analysys Mason made in its report is "substantially complete."
For example, Telecom has added four radio network controllers (RNCs) and has plans to have about eight RNCs in the network by June 2011 to address network capacity. For better coverage, the operator is adding cell sites and tower mounted amplifiers; and the operator has completed a software upgrade across the network, according to Reynolds.
"You know it's sobering, but it's heartening as well in the sense that there are some very clear lessons for us and Alcatel-Lucent to fix and we've fixed them," said Reynolds.
— Michelle Donegan, European Editor, Light Reading Mobile