Optical/IP Networks

Cisco Unplugged

For all its efforts to network the world, Cisco is having trouble keeping its online ordering systems going.

Yesterday, for the second time in nine days, Cisco's online order management systems went down, blocking customers from logging new orders or checking the status of older ones.

Cisco confirms that one outage occurred on Monday, Sept. 18; a customer tells Light Reading it lasted 20 hours. Yesterday's incident was relatively short, and the systems came back online before the end of the working day, Pacific time, according to a Cisco spokesman. (See Cisco Order Outage?)

Via email, the spokesman confirms the company having "some intermittent instability in our online ordering systems" but wouldn't divulge precise uptime/downtime data for either outage. He wasn't keen on describing the causes of the outages, either.

Some details came in a missive sent by Cisco's operations team to customers. (The spokesman has confirmed it as official.) It opens with: "Cisco is currently experiencing a system outage in its manufacturing applications on Cisco's CCX platform," CCX being the backbone for Cisco's order management tools and some related Web-based applications, all of which got taken out.

Cisco's IT department was "working on a manual process to push order management priority details out to the sites in order to continue production momentum," Cisco's message read, adding that the "customer-facing tools" were getting priority as far as which systems to fix first.

The whole Cisco manufacturing machine wasn't paralyzed, though. Some systems, such as those handling "shipping and packout," run on Oracle software and weren't affected by the CCX problem, according to Cisco's message.

Cisco's online systems are culled together with homegrown code and some vendor programs and tools, according to a spokesman.

One customer claims Cisco was telling customers not to talk to reporters, but the Cisco spokesman says no such edict came from the company officially.

— Craig Matsumoto, Senior Editor, Light Reading

Pete Baldwin 12/5/2012 | 3:39:34 AM
re: Cisco Unplugged I was wondering that myself, whether one or both outages might have come from a virus or worm, as opposed to something crashing "organically." Tuesday's outage was short, so it could have been a normal case of gremlins (something plugged into the wrong place, power lost to a certain building, beta code accidentally pushed to production....)

The 20-hour downtime must have been a doozy. Major hardware failure, maybe?
Belzebutt 12/5/2012 | 3:39:34 AM
re: Cisco Unplugged Did Cisco's network get hit by the IE VML exploit?
signmeup 12/5/2012 | 3:39:31 AM
re: Cisco Unplugged How about a major software upgrade that corrupted the database and required 20 hours to get the system restored and the data normalized?

This affected internal systems as well.

Sign In