x
Optical/IP

Net Up, Phones Busy in Blackout

In a moment of crisis, the Internet once again has proven to be more reliable than the telephone network.

As the Eastern U.S. and parts of Ontario today struggle to recover from a massive power outage, customers who had access to the Internet over phone lines or other connections were able to stay in touch by email and instant messaging, while many voice circuits were snarled with traffic overloads resulting in busy signals.

"I actually found out about the blackout before it hit any news sources via... one of my techs out in the field," writes Joel Perez, an IP engineer with Ntera, a facilities-based IP service provider headquartered in Florida, in an email today. "They lost all power and phone service. The only thing still running was his Internet connection at the PoP he was in at the moment!" Ntera lost several voice circuits in the Northeast, Perez says, but all Internet-based data links held firm.

All of the major U.S. Internet backbones from the 25 largest U.S. metropolitan areas continued to function normally following the blackout, according to Keynote Systems, an independent company that monitors Internet performance. Most major Websites remained accessible and at their normal performance levels. A few news sites, such as www.usatoday.com and www.cnn.com, had minor availability problems at the outset of the blackout, said Keynote.

Meanwhile, statements issued late yesterday by Bell Canada (NYSE/Toronto: BCE) and Verizon Communications Inc. (NYSE: VZ) boasted of minimal impact on the telecom networks. But in the case of many everyday customers in the New York area who could not get through on phone lines, the statements seemed to contradict reality.

Verizon's statement last night said wireline and wireless services were "performing normally" and "fully operational," but warned of high call volumes, which apparently gave many users a busy signal in place of a dial tone.

Repeated calls to Verizon produced only a busy signal this morning. Several Light Reading editors and family members, working from home in the New York City area, reported ongoing inability to communicate by wireline or cell phone.

Bell Canada's statement yesterday claimed its network "remained functional." This morning, though, the carrier issued another statement saying the network was functional and "99 percent of customers" had service. With 13 million lines in service, per Bell Canada's Website, that means at least 200,000 customers were without service.

A Bell Canada spokesman said he couldn't quantify the number of lines out, but asserted the carrier is "maintaing power to key facilities" for cell and wireline users.

The outage eliminated power in key cities in the public electrical grid from New York west to Ohio and Michigan and north into Ontario. It forced telecom networks into backup mode and drove up traffic on cellular and landline services, as distressed citizens everywhere took to the phones to call for help and contact relatives and coworkers.

The cause hasn't been determined, though speculation centers on a series of cascading circuit failures and subsequent overloads on the grid. The North American Electric Reliability Council is holding a press conference this morning to discuss the issues so far.

According to that organization's Website, the outage is the largest to hit the continent since widespread outages hit the Western U.S. and Canada, affecting 7.5 million customers, in July 1996.

The outage has raised a slew of questions about the vulnerabilities and reliability of telecom networks. Here is a sampling raised by posters to the mailing list North American Network Operators' Group (NANOG):


  • How is backup affected by who's in charge of collocation facilities? In one instance, a Canadian operator seemed to be suffering power loss because the building in which it was a tenant didn't have adequate backup in place.
  • What's backing up power? Verizon says it backed up power with diesel generators tripped in to power its call routing facilities. But some operators say Sonet gear in basements and other locations rely on battery backup, and they question how long that can last. Some say batteries can't be relied on for the full duration of eight to 12 hours promised by manufacturers.
  • How can telcos ensure batteries to back up numerous facilities stay charged? One poster says that during a past power outage, a Bell operating company successfully kept batteries running by ferrying generators by truck from site to site, charging batteries as it went.
  • What about the air conditioning? Sensitive computer and telecom gear needs air conditioning to avoid downtime. When the power is out, so is that life-giving coolant.
But perhaps the largest question in light of of the recent crisis: What about packet services vs. circuit? Given the Internet's recent performance during crisis, service providers might need to once again consider the advantages of packet-switched networks.

— Mary Jander, Senior Editor, and R. Scott Raynovich, US Editor, Light Reading

Page 1 / 7   >   >>
solver 12/4/2012 | 11:34:43 PM
re: Net Up, Phones Busy in Blackout While I agree with your observances, I have to raise a major disagreement with your broad conclusion.
The internet was more reliable in this instance, provided that you have backup power to your location of internet access. But when I go home, and my house or my MDU is completely dark as most residences were, I have no use for the NET that was up. However, I was able to rely on my (CO powereed) landline 10 out 10 times last night. I was cpmpletely unsuccessful in 5 attempts with my cell phone, which was expected.

I suppose the conclusion is that there's a inherent need for more capacity for the mobile network, while I have minimal complaints with the old POT.
And lastly, in a time of power crisis, the net is dependable as long as you can get to a location that has: plenty of internet access, power generator for backup, and plenty of diesel fuel.

dljvjbsl 12/4/2012 | 11:34:42 PM
re: Net Up, Phones Busy in Blackout
What's backing up power? Verizon says it backed up power with diesel generators tripped in to power its call routing facilities. But some operators say Sonet gear in basements and other locations rely on battery backup, and they question how long that can last. Some say batteries can't be relied on for the full duration of eight to 12 hours promised by manufacturers.


As far as I know all telecom equipment runs from the -48V battery. This comes from extremely large battery plants that take up entire basements of large COs. These batteries are kept charged from the mains or in case of mains failuress from generators.

I recall being in a CO once in which I was asked if I wanted to see the battery plant and a test of the duplicated gas turbine powered genrators which supplied back up. The sound of that turbine winding up was extremely impressive. It was a jet engine bolted to the floor of a basement room. You can imagine the feeling of power.
dljvjbsl 12/4/2012 | 11:34:42 PM
re: Net Up, Phones Busy in Blackout From the Toronto Star, regarding the response of the Canadian government. Note that the capital Ottawa was affected by the blackout.


Defence Minister John McCallum, speaking to Canadian and American reporters by teleconference from his Oakville [near Toronto] home, announced the power blackout that affected an area of 24,000 square kilometres was due to a fire at a nuclear plant.

``We are assured by the U.S. military that this fire did not involve an act of terrorism or sabotage," McCallum said, urging ``everyone to remain calm and patient" while power was being restored.

As McCallum continued, reporters scrambled to alert their media outlets and minutes later, an ABC reporter interrupted the minister to say the U.S. nuclear regulatory agency reported no fire at a nuclear plant anywhere in the U.S.

A clearly frustrated McCallum demanded his aide clarify the information, at which point communications director Randy Mylyk said the information indicated a ``power outage."


ian_cognito 12/4/2012 | 11:34:39 PM
re: Net Up, Phones Busy in Blackout Sentence 1: The Internet worked, the PHONES didn't

Sentence 2: Internet access using the PHONE LINES worked fine

What are you trying to say?

The bit-head vs bell-head thing is way-old, get over it.
Scott Raynovich 12/4/2012 | 11:34:37 PM
re: Net Up, Phones Busy in Blackout >Sentence 1: The Internet worked, the PHONES >didn't

>Sentence 2: Internet access using the PHONE >LINES worked fine

Actually, it doesn't say that... first sentence says "telephone network" and second sentence says "telephone lines." We were deliberate with the language because we knew of course that folks in our feisty reader base would "have issues" with it.

In general, I think "telephone network" covers everything from the copper endpoints through the switches in the voice system. Whereas "Lines" would just mean using the copper to get to another, say, packet-switched network. If you have better suggestions for handling this, I'd like to hear them. The semantics are always a little tricky.


rjmcmahon 12/4/2012 | 11:34:37 PM
re: Net Up, Phones Busy in Blackout As far as I know all telecom equipment runs from the -48V battery. This comes from extremely large battery plants that take up entire basements of large COs. These batteries are kept charged from the mains or in case of mains failuress from generators.

Won't modern technologies do a much better job? There are single board computers (SBCs), in the PC104 form factor, that can operate on less than 5Watts. Their processing power is respectable. Throw some solar battery backup systems into the mix, or some fuel cells, and we're off to the races ;-)

http://www.siliconsolar.com/
http://www.pc104.com
http://www.polyfuel.com

Diesel generators to power CPE devices seems like overkill as well as an environmental mess.
Raymand 12/4/2012 | 11:34:36 PM
re: Net Up, Phones Busy in Blackout rjmcmahon's disparaging the presence and use of diesel generators as emergency back up sources of power due to their impact on the environment is so infantile, pathetic and "mono-think-istic" (...Thank you. Thank you very much. I'd like to thank the academy, my language arts teacher,... and most of all my mom...) as to force me recommend to your employer that you be disbarred from contributing towards any business decision whatever; furthermore, I strongly suggest that your wife (or other as the case may be) take control of whatever joint financial resources may exist.


Cheers,

Everybody loves Raymand
grapsfan 12/4/2012 | 11:34:35 PM
re: Net Up, Phones Busy in Blackout > In general, I think "telephone network" covers > everything from the copper endpoints through
> the switches in the voice system.
> Whereas "Lines" would just mean using the
> copper to get to another, say, packet-switched
> network. If you have better suggestions for
> handling this, I'd like to hear them. The
> semantics are always a little tricky.


Well, IMO, the way to have handled it wasn't to try and take a pot shot at the legacy POTS phone network. From both descriptions, the network was behaving as designed. When POTS users grow larger than a 4:1 concentration on a given loop, users will get busy signals and "we're sorry" messages rather than being able to make a phone call. This is true whether there's a power blackout, Mother's Day, or the Cubs winning the World Series.

The information given in the article was fine...I'm just not sure why the "net good, phones suck" attitude crept through. I'll expect a similar but opposite article the next time a worm virus takes ISPs down, but the POTS network hums along.
skeptic 12/4/2012 | 11:34:35 PM
re: Net Up, Phones Busy in Blackout But perhaps the largest question in light of of the recent crisis: What about packet services vs. circuit? Given the Internet's recent performance during crisis, service providers might need to once again consider the advantages of packet-switched networks.
---------------------
I dont quite understand. The infrastructure
of circuit services stood up just fine to the
crisis. It acted as its supposed to for the
most part.

The *volume* of calls is what seems to have
exceeded capacity. And if you moved all of
those calls to packet-switched networks, you
might still end with the same situation.
Granted, however, that to the data network its
much less traffic.

The question on the circuit side is always
"are people willing to pay for the sort of
overprovisioned capacity it would take to
deal with a worst-case situation". The answer
has historically been "no".

dljvjbsl 12/4/2012 | 11:34:34 PM
re: Net Up, Phones Busy in Blackout
But perhaps the largest question in light of of the recent crisis: What about packet services vs. circuit? Given the Internet's recent performance during crisis, service providers might need to once again consider the advantages of packet-switched networks


The real question here is why LightReading doesn't know more about how the telephone network works. Perhaps the LightReading group should learn something about traffic engineering.
Page 1 / 7   >   >>
HOME
Sign In
SEARCH
CLOSE
MORE
CLOSE