& cplSiteName &

AT&T Begins Massive Battery Replacement

Phil Harvey
News Analysis
Phil Harvey
1/15/2008
50%
50%

After four equipment fires in two years, including a Christmas Day 2007 explosion in Wisconsin, AT&T Inc. (NYSE: T) says it is no longer comfortable with the batteries that provide backup power to thousands of its equipment cabinets in neighborhoods all over the U.S.

"Following incidents involving batteries used in AT&T U-verse network cabinets, the company is replacing 17,000 similar batteries, all manufactured by Avestor," writes an AT&T spokesman, in an email to Light Reading.

"Normally, we would work with a vendor to diagnose problems and develop solutions. We can't do that in this case because Avestor filed for bankruptcy in October 2006 and closed shortly thereafter. As a result, we have decided to move forward with the removal of all Avestor batteries as quickly as possible," the spokesman adds.

AT&T says it has no immediate guess as to how quickly it can replace all those batteries. The carrier also declined to speculate on the costs of such an endeavor.

A company spokesman does note, however, that AT&T stopped deploying Avestor batteries during the first quarter of 2007. The company announced a new battery supplier in July 2007. (See AT&T Eyes Batteries in Explosion Probe.)

Whatever the timing and cost, it won't be trivial, according to industry analyst Kermit Ross, principal of Millenium Marketing.

"It's no small task to change out the batteries in thousands of remote cabinets, especially when many of them are already powered up and handling working U-verse subscribers," Ross says. "It looks like a multimillion-dollar job, considering the cost of replacement batteries plus the labor to install them. They'll probably need to change the wiring to the batteries, too."

Bad battery background
AT&T's troubles with Avestor surfaced in October 2006, when an equipment cabinet exploded in a suburban Houston neighborhood, startling homeowners and ripping out a sizable chunk of fencing from one elderly couple's yard. (See AT&T Investigates DSLAM Explosion.)

In January 2007, another incident occurred just 20 miles away from the first one. This time the cabinet caught on fire, though that was quickly extinguished. However, the concerns about the safety of the batteries powering the cabinets kept growing. (See AT&T Confirms Second VRAD Fire.)

Following the first two Houston incidents, AT&T hired a leading scientific consulting firm to investigate the cause of the October 2006 equipment cabinet explosion and the January 2007 equipment fire.

The consulting firm, Exponent, concluded that the problems were caused by manufacturing defects, that the battery's safety features and overall design were "sound," and "concluded that the risk of hazardous failures with this battery is as low, if not lower, than the risk with alternative batteries." (See AT&T: Defect Caused VRAD Explosion.)

Light Reading requested a copy of the Exponent report from AT&T, but has not yet received one.

But wait... there's more
AT&T now admits it had two more incidents before the end of 2007 -- one small fire in an equipment cabinet and one cabinet explosion. The small fire occured in one of AT&T's cabinets near Cleveland, the carrier says, though it won't elaborate on the location.

But on Christmas Day, a cabinet at the corner of 64th Street and North Avenue in Wauwatosa, Wis., exploded and burned violently, according to city officials and a blog post on SaveAccess.

Local officials there used Light Reading's reporting on the earlier cabinet fires to learn how Avestor's lithium metal polymer batteries had been a factor in previous AT&T equipment cabinet incidents. (See Exclusive Photos: Fire to the Node.)

In the Wauwatosa fire, the main cabinet door was "displaced and landed about five to six feet southwest of the cabinet," according to Assistant Chief Jeffrey S. Hevey, Wauwatosa's fire marshal, who spoke with Light Reading on Monday.

Hevey reckons the cabinet door, which probably weighs 50 to 60 pounds, was blown off during the cabinet's explosion because the "bolts and rivets were sheared off cleanly," which suggests "a sudden, powerful displacement."

The cabinet that exploded in Wauwatosa sat next to a two-story office building with an exterior brick wall, so the building only suffered water damage (as the fire was extinguished) and some melted vinyl shutters as a result of the fire. Though companies transporting Avestor lithium metal polymer batteries are required to adhere to special hazardous material (hazmat) procedures, Hevey says the scene of the fire wasn't treated as a hazmat fire because, "basically, we didn't know much about it."

Wauwatosa residents report that AT&T moved quickly on the scene, clearing the debris and installing a new cabinet within seven hours of the explosion.

Now, all the controversy surrounding AT&T's cabinets has come full circle. The batteries, once deemed to be safe and sound by independent experts, have failed at least four times now, and the carrier has committed to continuing its removal and replacement of the 17,000 Avestor batteries spread out across its network.

On the one hand, AT&T pushed past the experts and is addressing a potentially huge public safety risk. "As we gained experience with these batteries, we felt that they no longer met our stringent safety and performance criteria," an AT&T spokesman told Light Reading on Monday.

On the other hand, Millennium Marketing's Ross says the carrier could have acted more quickly: "It would have been much less disruptive and costly to have addressed this problem a year ago, when it first cropped up."

— Phil Harvey, Editor, Light Reading

(14)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Page 1 / 2   >   >>
dsb
50%
50%
dsb,
User Rank: Light Beer
12/5/2012 | 3:49:58 PM
re: AT&T Begins Massive Battery Replacement
Another example of SBC being penny wise and pound foolish. This is the same company that takes the attitude that it's cheaper to pay million dollar fines than hire enough field techs to keep customer complaints at a reasonable level..
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:49:58 PM
re: AT&T Begins Massive Battery Replacement

What he is referring to (and this occurs at all the RBOCs and can occur at any ILEC) is fines based on percentage of outage time. There are requirements for uptimes and if they are not met, then the RBOC gets a fine. The same is true with held orders. If enough services are ordered but not delivered over a period of time, RBOCs and other ILECs can be fined.

This is nothing new and has been going on for years. He is implying that by hiring more people that AT&T could lower their fines (or eliminate them). That may or may not be true, but it is the implication.

seven
DCITDave
50%
50%
DCITDave,
User Rank: Light Beer
12/5/2012 | 3:49:58 PM
re: AT&T Begins Massive Battery Replacement
What do you mean?
DCITDave
50%
50%
DCITDave,
User Rank: Light Beer
12/5/2012 | 3:49:57 PM
re: AT&T Begins Massive Battery Replacement
A clarification: The batteries in question supply backup power, not main power. Just so everyone understands what we're talking about here.

ph
net-hed
50%
50%
net-hed,
User Rank: Light Beer
12/5/2012 | 3:49:56 PM
re: AT&T Begins Massive Battery Replacement

Phil, are you forced to share your news with other CMP sites?

http://www.eetimes.com/news/la...
DCITDave
50%
50%
DCITDave,
User Rank: Light Beer
12/5/2012 | 3:49:55 PM
re: AT&T Begins Massive Battery Replacement
no, i sent it to them as a courtesy. no need to have many writers covering the same thing.

unless it's apple, apparently.

ph.
Yao
50%
50%
Yao,
User Rank: Light Beer
12/5/2012 | 3:49:54 PM
re: AT&T Begins Massive Battery Replacement
Can anyone comment on what level of fine could it be, if the battery supplier did not go bankrupcy ? This could provide a hint on whether the risk was priced in correctly for anyone to do business with AT&T while the industry continues cost cutting and outsourcing
trzwuip
50%
50%
trzwuip,
User Rank: Light Beer
12/5/2012 | 3:49:53 PM
re: AT&T Begins Massive Battery Replacement
I lived in that neighborhood in Copperfield north of Houston. Back in 2000, SW Bell (Now AT&T) was using directional drilling to install fiber-optic lines for their then-new DSL service. Seems the contractor drilled through a natural gas line and then through a sewer line.

Unfortunately, they realized their mistake and pulled the bit out. The result was natural gas backing up into the sewer line causing the explosion of one house and severely damaging the the houses on either side. Between the broken glass and the fire, it was quite a mess!
yourehavinalarf
50%
50%
yourehavinalarf,
User Rank: Light Beer
12/5/2012 | 3:49:52 PM
re: AT&T Begins Massive Battery Replacement
Maybe they didn't renew their maintenance contract as they thought they could do it themselves and didn't maintain them properly.
Koolkid1935
50%
50%
Koolkid1935,
User Rank: Light Beer
12/5/2012 | 3:49:52 PM
re: AT&T Begins Massive Battery Replacement
Is that what caused that??? I remember that too, it isn't too far away from our house. I think the A/C turning on is what ignited the natural gas if memory serves me correctly.
Page 1 / 2   >   >>
Educational Resources
sponsor supplied content
Educational Resources Archive
Light Reading’s Upskill U is a FREE, interactive, online educational resource that delivers must-have education on themes that relate to the overall business transformation taking place in the communications industry.
NEXT COURSE
Wednesday, August 3, 1:00PM EDT
The Central Office Re-Architected as a Data Center
Guru Parulkar, Executive Director, Open Networking Research Center, Open Networking Lab
UPCOMING COURSE SCHEDULE
Wednesday, August 10, 1:00PM EDT
Telcos & Open Source 101
Phil Robb, Senior Technical Director, OpenDaylight
Friday, August 12, 1:00PM EDT
The Role of Open Source in NFV
Jim Fagan, Director, Cloud Practice, Telstra
Wednesday, August 17, 1:00PM EDT
Using Open Source for Data Centers and Cloud Services
Roz Roseboro, Senior Analyst, Heavy Reading
in association with:
From The Founder
Light Reading today starts a new voyage as part of a larger Enterprise.
Flash Poll
Live Streaming Video
Charting the CSP's Future
Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
LRTV Custom TV
Reinventing Operations for a Virtual, Software-Defined World

7|28|16   |   5:23   |   (0) comments


Heavy Reading Senior Analyst Jim Hodges speaks with Accenture's Larry Socher and Matt Anderson about what service providers must do to transform their business to get the benefits of SDN and NFV including: leveraging DevOps, introducing real-time OSS and implementing analytics.
Women in Comms Introduction Videos
Fujitsu Sales Leader Shares Lessons Learned

7|27|16   |   5:12   |   (0) comments


As Fujitsu's only female sales leader, Annie Bogue knows the importance of asking for what you want, being flexible (she's been relocated five times), keeping a meticulous calendar, 'leaning in,' working harder than everyone else around you, being aware and more.
Telecom Innovators Video Showcase
VeEX Test & Measurement Solutions

7|25|16   |   08:57   |   (0) comments


Cyrille Morelle, president and CEO of VeEX Inc., talks test and measurement with Light Reading's Steve Saunders at BCE 2016. This includes innovative products such as VeSion Cloud-Based platform for network monitoring; MTTplus Modular Test platform for Access, Business, Carrier Ethernet, Transport and Core services; and OPX-BOX+ for Fiber Optics.
LRTV Custom TV
VeEX: Live From BCE 2016

7|25|16   |   03:20   |   (0) comments


VeEX's Senior Director of Business Development, Perry Romano, explains how VeEX provides tools to help install, maintain, monitor and manage network infrastructure efficiently and effectively. The portfolio of products on display include the RXT-6000, MTTplus and TX300s.
LRTV Custom TV
Real-Time Telemetry & Analytics for Intelligent SDN Orchestration

7|25|16   |   03:09   |   (0) comments


Packet Design CEO Scott Sherwood discusses how real-time network telemetry and analytics are enabling a new breed of SDN orchestration applications.
From the Founder
The Russo Report: Driving Disruption

7|25|16   |   07:44   |   (0) comments


In the first episode of a four-part series, Light Reading Founder and CEO Steve Saunders and Calix President and CEO Carl Russo drive around town discussing the disruptive mega-changes in the communications industry and where hope lies for service providers to meet the escalating demands of the cloud.
LRTV Custom TV
NetScout: Maximizing Enterprise Cloud for Digital Transformation

7|20|16   |   04:53   |   (0) comments


Light Reading Editor Mitch Wagner talks to NetScout CMO Jim McNiel about maximizing the benefits of enterprise cloud and digital transformation while minimizing potential pitfalls with a proper monitoring and instrumentation strategy.
Women in Comms Introduction Videos
Ciena's VP Offers a Career Crash Course

7|20|16   |   4:14   |   (2) comments


How did Ciena's Vice President of Sales, Angela Finn, carve out her career path? Simple, she tells WiC. She stayed true to her company, customers and principles. She shares her advice for women on how to be authentic and credible, as well as for companies that want to make a real change to their culture and practices.
LRTV Custom TV
NFV in 2016: Part 2 – Climbing the Virtualization Maturity Curve

7|19|16   |   06:56   |   (0) comments


Many of the initial use case implementations are single-vendor and self-contained. The industry is still climbing the virtualization maturity curve, needing further clarity and stability in the NFV infrastructure (NFVi) and greater availability and choice of virtualized network functions (VNFs). Interoperability between NFVis and VNFs from different vendors ...
Telecom Innovators Video Showcase
Versa Networks' Kumar Mehta on SD-WAN Managed Services

7|19|16   |     |   (0) comments


In Silicon Valley, Steve Saunders sits down with Versa's Kumar Mehta for an interview focused on why service providers are building SD-WAN managed services, and how Versa's telco customers are innovating.
LRTV Custom TV
Juniper Networks & The Evolution of NFV

7|19|16   |   06:01   |   (0) comments


Senior Juniper Networks executives talk to Light Reading Founder & CEO Steve Saunders about NFV developments and the recent independent evaluation by test lab EANTC of Juniper's Cloud CPE solution.
LRTV Interviews
CenturyLink Goes Beyond Managed WiFi

7|19|16   |     |   (0) comments


CenturyLink's managed WiFi allows enterprises, such as retailers and resorts, to track guest WiFi usage in order to help them better communicate with customers.
Upcoming Live Events
September 13-14, 2016, The Curtis Hotel, Denver, CO
November 3, 2016, The Montcalm Marble Arch, London
November 30, 2016, The Westin Times Square, New York City
December 6-8, 2016,
May 16-17, 2017, Austin Convention Center, Austin, TX
All Upcoming Live Events
Infographics
Five of the Top 10 most targeted countries in Check Point Software Technologies' global Malware & Threat Index for Q1 2016 are in Africa.
Hot Topics
Verizon Sports Big Plans for Yahoo
Alan Breznick, Cable/Video Practice Leader, Light Reading, 7/26/2016
Light Reading Beyond
Steve Saunders, CEO and founder, Light Reading, 7/28/2016
Yahoo Signing Off in $4.83B Sale to Verizon
Mari Silbey, Senior Editor, Cable/Video, 7/25/2016
Ericsson Board Has Been Asleep at the Wheel – Consultant
Ray Le Maistre, Editor-in-chief, 7/25/2016
Facebook Gets Its Drone On
Ray Le Maistre, Editor-in-chief, 7/22/2016
Like Us on Facebook
Twitter Feed
BETWEEN THE CEOs - Executive Interviews
There's no question that, come 2020, 5G technology will turn the world's conception of what mobile networking is on its head. Within the world of 5G development, Dr. ...
I've enjoyed interviewing many interesting people since I rejoined Light Reading, but William A. "Bill" Owens certainly takes the biscuit, as we say where I come from.
Animals with Phones
Live Digital Audio

Our world has evolved through innovation from the Industrial Revolution of the 1740s to the information age, and it is now entering the Fourth Industrial Revolution, driven by technology. Technology is driving a paradigm shift in the way digital solutions deliver a connected world, changing the way we live, communicate and provide solutions. It can have a powerful impact on how we tackle some of the world’s most pressing problems. In this radio show, Caroline Dowling, President of Communications Infrastructure & Enterprise Computing at Flex, will join Women in Comms Director Sarah Thomas to discuss the impact technology has on society and how it can be a game-changer across the globe; improving lives and creating a smarter world. Dowling, a Cork, Ireland, native and graduate of Harvard Business School's Advanced Management Program, will also discuss her experience managing an international team focused on innovation in an age of high-speed change.