& cplSiteName &
Comments
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Page 1 / 2   >   >>
f_goldstein
f_goldstein
1/4/2019 | 9:28:43 AM
Re: Reminiscent of TARP storms of old
Like many bugs in complex software, it is hard to test for and you don't know it's there until it's too late. Especially when it's triggered by a hardware failure. There's a hardware failure mode in some routers wherein the card stops relaying MPLS and other traffic (its job) but does maintain physical connectivity and its IS-IS daemon dutifully reports that the link is up. In that case the black hole it creates is relatively easy to locate. GMPLS, like MPLS and IP, depends on some underlying route determination code. And apparently the bad packets knocked out circuits previously set up. Here, it seems as if a packet of death, with no address, was allowed to propagate. The vendor probably didn't test for that since it's not supposed to happen. I'm guessing that a misdesign in the vendor code somewhere relayed, rather than forwarded, that packet, perhaps treating its non-address as  a broadcast address. That would really be, uh, hilarious.

But at this point we're just guessing based on limited information.
Sterling Perrin
Sterling Perrin
1/4/2019 | 8:49:30 AM
Re: Reminiscent of TARP storms of old
f_goldstein,

Your explanation seems the most reasonable of the ones I've seen so far - i.e., the explanation does fit the cryptic CTL commentary. IF this is the explanation, one thing that puzzles me is that GMPLS is well-entrenched in networks and has been for more than a decade. So why has this never happened before? (I'm not aware of a GMPLS breakdown like this at any time in the past.)

 

Sterling
Phil Harvey
Phil Harvey
1/3/2019 | 6:48:34 PM
Re: Reminiscent of TARP storms of old
CTL said it "can't confirm its authenticity and don't have any additional information to share."

Which is not the same as, that's a fake, we didn't write it.
Phil Harvey
Phil Harvey
1/3/2019 | 6:02:55 PM
Re: Reminiscent of TARP storms of old
I'll have to ask CTL but it has some of the exact phrases that they sent me in their email replies to my questions. 
f_goldstein
f_goldstein
1/3/2019 | 5:59:09 PM
Re: Reminiscent of TARP storms of old
GMPLS comes to mind, because they were losing optical streams, and that's the usual approach to treating optical streams like BESQR cat videos. I am guessing that "third party" simply means the hardware vendor, not a real third party. There's an interesting alleged "outage report" on comp.dcom.telecom.
Phil Harvey
Phil Harvey
1/3/2019 | 5:40:04 PM
Re: Reminiscent of TARP storms of old
And, if so, do you interpret CTL's statement of "third-party network management card" as someone who built a protocol stack for the systems vendor? 
Phil Harvey
Phil Harvey
1/3/2019 | 5:38:52 PM
Re: Reminiscent of TARP storms of old
So you think it was GMPLS signaling between optical transport devices?
f_goldstein
f_goldstein
1/3/2019 | 2:54:04 PM
Re: Reminiscent of TARP storms of old
The outage impacted optical services, not just packet service, but was caused by packets. That points to a control plane on an optical network that has something more elaborte than simple mangement commands. The packet of death didn't just knock out change-control, it knocked out circuits. What could that be?

I'm guessing that this was a GMPLS failure. That takes all of the brokenness of the IP protocol suite and puts it in charge of the underlying optical layer. It's what comes from folks who don't come from telecom, and who think the whole world will forever be IP and that IP is somehow infallible heavenly writ. The higher layer routing protocol's job is to route around physical failures. Put the physical layer under its control and of course hilarity ensues.

Surely somebody here is willing to name CTL's major optical vendor.
brooks7
brooks7
1/3/2019 | 10:27:04 AM
Re: Reminiscent of TARP storms of old
@Duh!,

 

And that is why a failure would go so badly.....

seven

 
Duh!
Duh!
1/2/2019 | 10:49:57 PM
Re: Reminiscent of TARP storms of old
X.25 is the absolutely last protocol suite I would ever associate with packet storms. Heavyweight flow control was one of it's main architectural principles.
Page 1 / 2   >   >>


Featured Video
Upcoming Live Events
October 1-2, 2019, New Orleans, Louisiana
October 10, 2019, New York, New York
October 22, 2019, Los Angeles, CA
November 5, 2019, London, England
November 7, 2019, London, UK
November 14, 2019, Maritim Hotel, Berlin
December 3, 2019, New York, New York
December 3-5, 2019, Vienna, Austria
March 16-18, 2020, Embassy Suites, Denver, Colorado
May 18-20, 2020, Irving Convention Center, Dallas, TX
All Upcoming Live Events
Partner Perspectives - content from our sponsors
Why Are Governments Around the World Subsidizing 5G?
By Paul Zhou, FromGeek.com, for Huawei
Edge Computing, the Next Great IT Revolution
By Rajesh Gadiyar, Vice President & CTO, Network & Custom Logic Group, Intel Corp
Innovations in Home Media Terminals for the Upcoming 5G Era
By Tang Wei, Vice President, ZTE Corporation
All Partner Perspectives