Mobile security

Mobile Ops Lose $15B Yearly to Network Outages

Mobile operators suffer from an average of five network outages or degradations that impact subscribers each year, costing them around $15 billion annually, according to new Heavy Reading research.

Put another way, that's about one outage every other month. More than 80 percent of those outages affect just one or a subset of networks or services. But given that operators are losing customers and spending an average of 1.5 percent of their annual revenues -- with some as high as 5 percent -- trying to rectify outages, the cause for concern is understandable. (See Outage Outrage.)

Heavy Reading 's new report, Mobile Network Outages & Service Degradations: A Heavy Reading Survey Analysis, looks at what causes outages, why they happen, and what can be done. These are issues that are becoming more pressing for operators as they roll out LTE networks for which expectations are high. (See LTE Brings Myriad Security Concerns and Can Mobile Networks Cope?)

According to Heavy Reading senior analyst and report author Patrick Donegan, the most common causes are physical link failures, network congestion or overloading, and network failures. A chip might melt if the air conditioning breaks, the power could go out, or equipment could break.

What surprised Donegan, who has been studying the mobile security market, was that malicious attacks didn't feature prominently as a cause of network outages. In fact, most operators ranked it last on the list of causes. That may not mean those attacks aren't happening -- operators, even those with network security experts on staff, just don't always have network visibility that is granular enough to discern every root cause.

"Part of the reason for that is many operators actually have little or no visibility of the malicious traffic in the network," Donegan says. "When they do have incidents, often they are not actually aware if it may have been a malicious attack that caused it."

Applications on the network trigger a lot of signaling traffic, and the operator can't always distinguish if that traffic is being generated by a benign form of behavior or protocols or whether it's malicious traffic caused by an outside attacker, Donegan adds. This is something the operators surveyed readily admitted. One commented:

We often have no clear understanding of outages and degradations and what causes them, and our RAN vendors often don't understand either.

To some degree, occasional outages are inevitable when operating a network, but Donegan says there are things the operators can do to minimize and understand the impact. First and foremost is to employ high-caliber people at both the operator and their primary infrastructure vendors and then make sure the two are communicating well. Reaction times can be unnecessarily long when the two are inadequately trained or not communicating.

"There may be one kink in that chain of communication, and you have the potential to cause a network incidence," Donegan says.

Donegan and Heavy Reading will be discussing more ways to gain insight into network outages and prevent malicious attacks at the upcoming Mobile Network Security Strategies, a Light Reading Live event that takes place on December 5, 2013 at the Westin Times Square Hotel in New York City. For more information, or to register, click here.

— Sarah Reedy, Senior Editor, Light Reading

JamesDennis 11/22/2013 | 5:37:33 PM
Outages Having been through numerous full mobile operator network outages the common denominator has been a cavalier attitiude to redundancy.

The cost benefit analysis of full redundancy vs. a few hours / days downtime a year has been done and operators are willing to take the risk .

No-one operates networks in the old school every element  has its counterpart fashion.

When it comes  to the heavy equipment, virtualised servers are not an equivalent, cloud soluitions are a diversion, one has to have some basics, bandwidth, signalling capability,  power and  physical space somewhere. So the problem is not going away anyday soon.

It more likely to get worse, as the production values decline for mobile services.

The upside is and there is one, goes like this, despite the steady decline in availability, subscriptions and traffic charges are a fraction of what they once were.

Subscribers should take on the mantle of availability for  themselves by using mulitiple  suppliers. Its a consumer choice, abeit not one that operators are likely to shout about.

Its all the customers own fault for choosing to be with the network that went down -
                (dyed in the wool operations ideology comes out)



RitchBlasi 11/19/2013 | 3:56:08 PM
Network Outages Before I get into this, let me fess up that my last three years I worked at a major mobile operator, as executive communications director for the network and technology groups.  That said, I was never a real geek but knew enough to be dangerous...and then some.   

Outages come in all shape and sizes, and can be caused by a variety of things – from a single sector of a cell site going down due to a bad antenna to a beaver eating through some fiber backhaul.  Yes, I've heard them all. 

While monitoring detailed info on networks is a challenge – like differentiating a customer watching porn on their iPad from a someone downloading a 200MB Powerpoint presentation – these guys pretty much know what's happening.  When I worked there we had a reporting tool that alerted me (and everyone else who wanted/needed to be notified) to outages around the country, and what level outage it was – tier 1,2 or 3.  I also was given an ETC (estimated time of completion) for the outage, what the root cause was, and what was done to get it fixed.  If it affected a lot of customers or a good sized geographical area, we often notified local media. 

So there is actually a lot of monitoring going on.  If you ever have the chance to get to the Network Operations Center for one of the carriers, do it.  It's like walking into NASA's Control Center. 

Anyway, I digress somewhat.  I took as someone else am interested in what the report includes.  When mobile carriers spend between $5-8 billion a year on network expansion and capacity, that $15 billion/year strictly for outages seems high to me – even if shared by all the carriers.  Unless it includes deployments of COWs, more personnel, upgrade of equipment, building additional towers so that cell sites are not overtaxed and a bunch of other stuff that might be part of the annual Capex spend. 

I'm sure it will be a good read.
MarkC73 11/18/2013 | 2:50:15 AM
Re: Communicating outages I'd sure like to see the break down on that $15B.  Also, in my experience the number one thing to cause failures for Mobile Operators is power.  Or maybe that's just in my neck of the woods.  I guess we have to define outage.  Lastly, if your service is out and assuming you can't call (because of the outage), I guess there's the radio.  Depending on how local, your local stations are.  But you are right, unless its big enough where the carriers know they'll have to release a statement, they pretty much don't post anything until after they understand the situation and are moving to fix it.  Probably too little to late for most folks.
pdonegan67 11/15/2013 | 11:12:18 AM
Re: Communicating outages 25% off? That's better than the 1,000 free texts an awful lot of mobile operators offer up as "compensation" for an outage.
Sarah Thomas 11/15/2013 | 11:08:05 AM
Re: Communicating outages It's not a network operator, but deals site Living Social just had a 40-hour outage. To apologize, it's offering 25% off the deals on its site. This is another good lesson -- offering deals after outages is a good way to appease your customers... http://allthingsd.com/20131114/livingsocial-to-run-discount-promotions-in-wake-of-40-hour-outage/
Sarah Thomas 11/15/2013 | 10:24:35 AM
Re: Communicating outages That is a good point, Dan. Patrick said that they often don't know if it's the case, but even if they do, I'm sure it's not something they'd want to publicize. More communication -- even just a "we're working on it" -- notice would help though.
Sarah Thomas 11/15/2013 | 10:23:27 AM
Re: Communicating outages Thanks, Marson. I agree that testing and monitoring the network will be crucial. Do you expect hetnet deployments to have the same growing pains at early LTE launches (like Verizon experienced)? Seems like there is more potential for issues with handoff and more complexity, as you noted.
MarsonJDSU 11/15/2013 | 7:54:57 AM
Re: Communicating outages "Sarah, interesting story. It's often not the complete network outages that frustrates users, but those unexplained intermittent periods of poor performance. As networks become even more complex - with the build out of small cells, het-nets and/or inclusion of virtualized components, etc. -  it just makes a stronger case for more intelligent monitoring and visibility solutions that can provide granular, app-level visibility. Increasingly mobile is becoming the primary source of network access - monitoring and assurance solutions that help operators identify and fix network issues quickly become a real competitive advantage. Of course, I suppose I am a bit biased but at the same time I get virtually all my information from my mobile phone these days, not my TV."
DOShea 11/14/2013 | 9:17:45 PM
Re: Communicating outages I think they rank attacks low on their list of network outage causes because they might not want customers to know the real cause. Likewise, they might not communicate much information about an outage to customers because they might not want to draw attention to it--that would be a foolish strategy, but I wouldn't rule it out.
Sarah Thomas 11/14/2013 | 3:42:38 PM
Communicating outages I've never experienced a long mobile network outage (degradation, sure, a lot), but I have to my TV service and power. The most frustrating part is being in the dark (literally, sometimes) about what's going on and when it will be resolved. I think service providers also need to do a better job of communicating what is going on to their customers on another channel than the network that is down.
Sign In