As the Eastern U.S. and parts of Ontario today struggle to recover from a massive power outage, customers who had access to the Internet over phone lines or other connections were able to stay in touch by email and instant messaging, while many voice circuits were snarled with traffic overloads resulting in busy signals.
"I actually found out about the blackout before it hit any news sources via... one of my techs out in the field," writes Joel Perez, an IP engineer with Ntera, a facilities-based IP service provider headquartered in Florida, in an email today. "They lost all power and phone service. The only thing still running was his Internet connection at the PoP he was in at the moment!" Ntera lost several voice circuits in the Northeast, Perez says, but all Internet-based data links held firm.
All of the major U.S. Internet backbones from the 25 largest U.S. metropolitan areas continued to function normally following the blackout, according to Keynote Systems, an independent company that monitors Internet performance. Most major Websites remained accessible and at their normal performance levels. A few news sites, such as www.usatoday.com and www.cnn.com, had minor availability problems at the outset of the blackout, said Keynote.
Meanwhile, statements issued late yesterday by Bell Canada (NYSE/Toronto: BCE) and Verizon Communications Inc. (NYSE: VZ) boasted of minimal impact on the telecom networks. But in the case of many everyday customers in the New York area who could not get through on phone lines, the statements seemed to contradict reality.
Verizon's statement last night said wireline and wireless services were "performing normally" and "fully operational," but warned of high call volumes, which apparently gave many users a busy signal in place of a dial tone.
Repeated calls to Verizon produced only a busy signal this morning. Several Light Reading editors and family members, working from home in the New York City area, reported ongoing inability to communicate by wireline or cell phone.
Bell Canada's statement yesterday claimed its network "remained functional." This morning, though, the carrier issued another statement saying the network was functional and "99 percent of customers" had service. With 13 million lines in service, per Bell Canada's Website, that means at least 200,000 customers were without service.
A Bell Canada spokesman said he couldn't quantify the number of lines out, but asserted the carrier is "maintaing power to key facilities" for cell and wireline users.
The outage eliminated power in key cities in the public electrical grid from New York west to Ohio and Michigan and north into Ontario. It forced telecom networks into backup mode and drove up traffic on cellular and landline services, as distressed citizens everywhere took to the phones to call for help and contact relatives and coworkers.
The cause hasn't been determined, though speculation centers on a series of cascading circuit failures and subsequent overloads on the grid. The North American Electric Reliability Council is holding a press conference this morning to discuss the issues so far.
According to that organization's Website, the outage is the largest to hit the continent since widespread outages hit the Western U.S. and Canada, affecting 7.5 million customers, in July 1996.
The outage has raised a slew of questions about the vulnerabilities and reliability of telecom networks. Here is a sampling raised by posters to the mailing list North American Network Operators' Group (NANOG):
- How is backup affected by who's in charge of collocation facilities? In one instance, a Canadian operator seemed to be suffering power loss because the building in which it was a tenant didn't have adequate backup in place.
- What's backing up power? Verizon says it backed up power with diesel generators tripped in to power its call routing facilities. But some operators say Sonet gear in basements and other locations rely on battery backup, and they question how long that can last. Some say batteries can't be relied on for the full duration of eight to 12 hours promised by manufacturers.
- How can telcos ensure batteries to back up numerous facilities stay charged? One poster says that during a past power outage, a Bell operating company successfully kept batteries running by ferrying generators by truck from site to site, charging batteries as it went.
- What about the air conditioning? Sensitive computer and telecom gear needs air conditioning to avoid downtime. When the power is out, so is that life-giving coolant.
— Mary Jander, Senior Editor, and R. Scott Raynovich, US Editor, Light Reading