Light Reading

AT&T Begins Massive Battery Replacement

Phil Harvey
News Analysis
Phil Harvey
1/15/2008
50%
50%

After four equipment fires in two years, including a Christmas Day 2007 explosion in Wisconsin, AT&T Inc. (NYSE: T) says it is no longer comfortable with the batteries that provide backup power to thousands of its equipment cabinets in neighborhoods all over the U.S.

"Following incidents involving batteries used in AT&T U-verse network cabinets, the company is replacing 17,000 similar batteries, all manufactured by Avestor," writes an AT&T spokesman, in an email to Light Reading.

"Normally, we would work with a vendor to diagnose problems and develop solutions. We can't do that in this case because Avestor filed for bankruptcy in October 2006 and closed shortly thereafter. As a result, we have decided to move forward with the removal of all Avestor batteries as quickly as possible," the spokesman adds.

AT&T says it has no immediate guess as to how quickly it can replace all those batteries. The carrier also declined to speculate on the costs of such an endeavor.

A company spokesman does note, however, that AT&T stopped deploying Avestor batteries during the first quarter of 2007. The company announced a new battery supplier in July 2007. (See AT&T Eyes Batteries in Explosion Probe.)

Whatever the timing and cost, it won't be trivial, according to industry analyst Kermit Ross, principal of Millenium Marketing.

"It's no small task to change out the batteries in thousands of remote cabinets, especially when many of them are already powered up and handling working U-verse subscribers," Ross says. "It looks like a multimillion-dollar job, considering the cost of replacement batteries plus the labor to install them. They'll probably need to change the wiring to the batteries, too."

Bad battery background
AT&T's troubles with Avestor surfaced in October 2006, when an equipment cabinet exploded in a suburban Houston neighborhood, startling homeowners and ripping out a sizable chunk of fencing from one elderly couple's yard. (See AT&T Investigates DSLAM Explosion.)

In January 2007, another incident occurred just 20 miles away from the first one. This time the cabinet caught on fire, though that was quickly extinguished. However, the concerns about the safety of the batteries powering the cabinets kept growing. (See AT&T Confirms Second VRAD Fire.)

Following the first two Houston incidents, AT&T hired a leading scientific consulting firm to investigate the cause of the October 2006 equipment cabinet explosion and the January 2007 equipment fire.

The consulting firm, Exponent, concluded that the problems were caused by manufacturing defects, that the battery's safety features and overall design were "sound," and "concluded that the risk of hazardous failures with this battery is as low, if not lower, than the risk with alternative batteries." (See AT&T: Defect Caused VRAD Explosion.)

Light Reading requested a copy of the Exponent report from AT&T, but has not yet received one.

But wait... there's more
AT&T now admits it had two more incidents before the end of 2007 -- one small fire in an equipment cabinet and one cabinet explosion. The small fire occured in one of AT&T's cabinets near Cleveland, the carrier says, though it won't elaborate on the location.

But on Christmas Day, a cabinet at the corner of 64th Street and North Avenue in Wauwatosa, Wis., exploded and burned violently, according to city officials and a blog post on SaveAccess.

Local officials there used Light Reading's reporting on the earlier cabinet fires to learn how Avestor's lithium metal polymer batteries had been a factor in previous AT&T equipment cabinet incidents. (See Exclusive Photos: Fire to the Node.)

In the Wauwatosa fire, the main cabinet door was "displaced and landed about five to six feet southwest of the cabinet," according to Assistant Chief Jeffrey S. Hevey, Wauwatosa's fire marshal, who spoke with Light Reading on Monday.

Hevey reckons the cabinet door, which probably weighs 50 to 60 pounds, was blown off during the cabinet's explosion because the "bolts and rivets were sheared off cleanly," which suggests "a sudden, powerful displacement."

The cabinet that exploded in Wauwatosa sat next to a two-story office building with an exterior brick wall, so the building only suffered water damage (as the fire was extinguished) and some melted vinyl shutters as a result of the fire. Though companies transporting Avestor lithium metal polymer batteries are required to adhere to special hazardous material (hazmat) procedures, Hevey says the scene of the fire wasn't treated as a hazmat fire because, "basically, we didn't know much about it."

Wauwatosa residents report that AT&T moved quickly on the scene, clearing the debris and installing a new cabinet within seven hours of the explosion.

Now, all the controversy surrounding AT&T's cabinets has come full circle. The batteries, once deemed to be safe and sound by independent experts, have failed at least four times now, and the carrier has committed to continuing its removal and replacement of the 17,000 Avestor batteries spread out across its network.

On the one hand, AT&T pushed past the experts and is addressing a potentially huge public safety risk. "As we gained experience with these batteries, we felt that they no longer met our stringent safety and performance criteria," an AT&T spokesman told Light Reading on Monday.

On the other hand, Millennium Marketing's Ross says the carrier could have acted more quickly: "It would have been much less disruptive and costly to have addressed this problem a year ago, when it first cropped up."

— Phil Harvey, Editor, Light Reading

(14)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Page 1 / 2   >   >>
dsb
50%
50%
dsb,
User Rank: Light Beer
12/5/2012 | 3:49:58 PM
re: AT&T Begins Massive Battery Replacement
Another example of SBC being penny wise and pound foolish. This is the same company that takes the attitude that it's cheaper to pay million dollar fines than hire enough field techs to keep customer complaints at a reasonable level..
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:49:58 PM
re: AT&T Begins Massive Battery Replacement

What he is referring to (and this occurs at all the RBOCs and can occur at any ILEC) is fines based on percentage of outage time. There are requirements for uptimes and if they are not met, then the RBOC gets a fine. The same is true with held orders. If enough services are ordered but not delivered over a period of time, RBOCs and other ILECs can be fined.

This is nothing new and has been going on for years. He is implying that by hiring more people that AT&T could lower their fines (or eliminate them). That may or may not be true, but it is the implication.

seven
DCITDave
50%
50%
DCITDave,
User Rank: Light Beer
12/5/2012 | 3:49:58 PM
re: AT&T Begins Massive Battery Replacement
What do you mean?
DCITDave
50%
50%
DCITDave,
User Rank: Light Beer
12/5/2012 | 3:49:57 PM
re: AT&T Begins Massive Battery Replacement
A clarification: The batteries in question supply backup power, not main power. Just so everyone understands what we're talking about here.

ph
net-hed
50%
50%
net-hed,
User Rank: Light Beer
12/5/2012 | 3:49:56 PM
re: AT&T Begins Massive Battery Replacement

Phil, are you forced to share your news with other CMP sites?

http://www.eetimes.com/news/la...
DCITDave
50%
50%
DCITDave,
User Rank: Light Beer
12/5/2012 | 3:49:55 PM
re: AT&T Begins Massive Battery Replacement
no, i sent it to them as a courtesy. no need to have many writers covering the same thing.

unless it's apple, apparently.

ph.
Yao
50%
50%
Yao,
User Rank: Light Beer
12/5/2012 | 3:49:54 PM
re: AT&T Begins Massive Battery Replacement
Can anyone comment on what level of fine could it be, if the battery supplier did not go bankrupcy ? This could provide a hint on whether the risk was priced in correctly for anyone to do business with AT&T while the industry continues cost cutting and outsourcing
trzwuip
50%
50%
trzwuip,
User Rank: Light Beer
12/5/2012 | 3:49:53 PM
re: AT&T Begins Massive Battery Replacement
I lived in that neighborhood in Copperfield north of Houston. Back in 2000, SW Bell (Now AT&T) was using directional drilling to install fiber-optic lines for their then-new DSL service. Seems the contractor drilled through a natural gas line and then through a sewer line.

Unfortunately, they realized their mistake and pulled the bit out. The result was natural gas backing up into the sewer line causing the explosion of one house and severely damaging the the houses on either side. Between the broken glass and the fire, it was quite a mess!
yourehavinalarf
50%
50%
yourehavinalarf,
User Rank: Light Beer
12/5/2012 | 3:49:52 PM
re: AT&T Begins Massive Battery Replacement
Maybe they didn't renew their maintenance contract as they thought they could do it themselves and didn't maintain them properly.
Koolkid1935
50%
50%
Koolkid1935,
User Rank: Light Beer
12/5/2012 | 3:49:52 PM
re: AT&T Begins Massive Battery Replacement
Is that what caused that??? I remember that too, it isn't too far away from our house. I think the A/C turning on is what ignited the natural gas if memory serves me correctly.
Page 1 / 2   >   >>
Educational Resources
sponsor supplied content
Educational Resources Archive
Flash Poll
From The Founder
Last week I dropped in on "Hotlanta," Georgia to moderate Light Reading's inaugural DroneComm conference – a unique colloquium investigating the potential for drone communications to disrupt the world's telecom ecosystem. As you will see, it was a day of exploration and epiphany...
LRTV Documentaries
Cable Eyeing SDN for Headend, Home Uses

5|26|15   |   05:57   |   (1) comment


CableLabs is looking at virtualizing CMTS and CCAP devices in the headend, as well as in-home devices, says CableLabs' Karthik Sundaresan.
LRTV Documentaries
Verizon's Emmons: SDN Key to Cost-Effective Scaling

5|22|15   |   03:53   |   (0) comments


For Verizon and other network operators to ramp up available bandwidth cost effectively, they need to move to SDN and agree on how to do that.
LRTV Documentaries
Lack of Universal SDN a Challenge

5|21|15   |   04:51   |   (3) comments


Heavy Reading Analyst Sterling Perrin talks about how uncertainty about SDN standards and approaches may be slowing deployment.
LRTV Custom TV
Steve Vogelsang Interview: Carrier SDN

5|20|15   |   05:02   |   (0) comments


Sterling Perrin speaks to Steve Vogelsang, Alcatel-Lucent CTO for IP Routing & Transport business, about the new Carrier SDN-enabling Network Services Platform and the operator challenges it solves.
LRTV Custom TV
Carrier SDN: On-Demand Networks for an On-Demand World

5|20|15   |   20:52   |   (0) comments


Steve Vogelsang, Alcatel-Lucent CTO for IP Routing & Transport business, talks about requirements and benefits of Carrier SDN during the keynote address at the Light Reading Carrier SDN event May 2015.
LRTV Documentaries
The Security Challenge of SDN

5|19|15   |   02:52   |   (0) comments


CenturyLink VP James Feger discusses concerns that virtualization could create new vulnerabilities unless network operators build in safeguards.
LRTV Custom TV
NFV Elasticity – Highly Available VNF Scale-Out Architectures for the Mobile Edge

5|18|15   |   5:50   |   (0) comments


Peter Marek and Paul Stevens from Advantech Networks and Communications Group talk about their NFV Elasticity initiative and the company's latest platforms for deploying virtual network functions at the edge of the network. Packetarium XL and the new Versatile Server Module: 'designed to reach parts of the network that other servers cannot reach.'
LRTV Huawei Video Resource Center
Bay Area Spark Meetup 2015

5|14|15   |   3:54   |   (0) comments


Developed in 2009, Apache Spark is a powerful open source processing engine built around speed, ease of use and sophisticated analytics. This spring, Huawei hosted a meetup for Spark developers and data scientists in Santa Clara, California. Light Reading spoke with organizers and attendees about Huawei's code contributions and long-term commitment to Spark.
LRTV Custom TV
The Transport SDN Buzz

5|12|15   |   06:01   |   (1) comment


Sterling Perrin, senior analyst at Heavy Reading, speaks with Peter Ashwood-Smith of Huawei and Guru Parulkar of ON.Lab about the evolution of transport SDN and the integration of technologies.
LRTV Custom TV
Next-Generation CCAP: Cisco cBR-8 Evolved CCAP

5|5|15   |   04:49   |   (0) comments


John Chapman, Cisco's CTO of Cable Access Business Unit and Cisco Fellow, explained the innovation design of Cisco's cBR-8, the industry's first Evolved CCAP, including DOCSIS 3.1 design from ground-up, distributed CCAP with Remote PHY and path to virtualization. Cisco's cBR-8 Evolved CCAP is the platform that will last through the transitions.
LRTV Custom TV
Meeting the Demands of Bandwidth & Service Group Growth

5|1|15   |   5:35   |   (0) comments


Jorge Salinger, Comcast's Vice President of Access Architecture, explains how DOCSIS 3.1 and multi-service CCAP can meet the demands of the bandwidth and service group growth.
LRTV Custom TV
DOCSIS 3.1: Transforming Cable From Hardware-Defined Network to Software-Defined Network

4|29|15   |   03:48   |   (0) comments


John Chapman, Cisco's CTO of Cable Access Business Unit and Cisco Fellow, explains how DOCSIS 3.1 can transform cable HFC network to a more agile software-defined network.
Upcoming Live Events
June 8, 2015, Chicago, IL
June 9, 2015, Chicago, IL
June 9-10, 2015, Chicago, IL
June 10, 2015, Chicago, IL
September 29-30, 2015, The Westin Grand Müchen, Munich, Germany
October 6, 2015, The Westin Peachtree Plaza, Atlanta, GA
October 6, 2015, Westin Peachtree Plaza, Atlanta, GA
All Upcoming Live Events
Infographics
Procera has gathered facts, stats and customer experience feedback from a survey of 540 users from across the globe.
Hot Topics
10 Alternate Uses for Tablets
Eryn Leavens, Copy Desk Editor, 5/22/2015
Bidding War for TWC Looks Likelier
Alan Breznick, Cable/Video Practice Leader, 5/22/2015
Charter Seals Deals for TWC, Bright House
Mari Silbey, Senior Editor, Cable/Video, 5/26/2015
Eurobites: Alcatel-Lucent Trials 400G in Czech Republic
Paul Rainford, Assistant Editor, Europe, 5/26/2015
Comcast Targets 6 New Gigabit Markets
Mari Silbey, Senior Editor, Cable/Video, 5/21/2015
Like Us on Facebook
Twitter Feed
BETWEEN THE CEOs - Executive Interviews
On May 29th 10 AM ET, Steve Saunders, founder and CEO of Light Reading, will be drilling into the "pains and gains" of NFV with Saar Gillai, SVP & GM for NFV at Hewlett-Packard Co. (NYSE: HPQ) (HP). He has defined a four-step NFV model describing a sequence of technology innovation. It's a must-read doc for any network architect looking to get to grips with their NFV migration strategy. Join us for the interview, and the chance to ask Saar your NFV questions directly!
With 200 customers in 60 countries, Stockholm-based Net Insight has carved out a solid leadership position in one of the hottest vertical markets going in comms right now: helping service providers and broadcasters deliver video and other multimedia traffic over IP networks. How has Net Insight managed to achieve this success in the face of immense competition from the industry giants?
My ongoing interview tour of the leading minds of the telecom industry recently took me to Richardson, Texas, where I met with Rod Naphan, CTO and SVP, Solutions, ...
Cats with Phones
Too Fluffy to Talk Click Here
Elmer found that his bountiful fur got in the way of meaningful conversation.