Network Operators List Their Peeves

The ears of a few well known networking companies must have been burning this week, as subscribers to the mailing list of the North American Network Operators' Group (NANOG) made their choices for the world's worst design decisions in networking gear.

These folk have plenty to complain about. After all, many of them are the guys'n'gals who get called out at night to fix your ISP connection. Their beefs (beeves?) center on features of networking gear that interfere with the efficient completion of their appointed rounds.

Examples include modules that can't be slotted into or out of routers easily; cable adapters that don't fit the equipment; sharp edges that cut engineers' hands; handles that don't work...

Here's a sampling of the choicer bits:

  • "Why did Cisco not include side handles on the 12000 chassis? It's a heavy chassis, and I can imagine how many techs have thrown out their back moving that chassis around." (NOTE: Subsequently, the thread moved on to how the scissors-jack included with the product could lift a house, or double as an office coffee-table.)
  • "Personally my issues are console-cable related: is there a benefit to the HUGE variety of console pinouts used by the various hardware vendors?"
  • "Could people just pick ONE pinout and connector and stick with it? Please!"
  • "I can't stand it when I sit down and find the keyboard in front of me has moved the 'backslash' key. It drives me crazy and prompts me to find a real keyboard right away to work with."
  • "RJ45 connecters that have a rubber hood over the release. Grrrrr!"
  • "The little clippy widgets (looks kind of like @) on some oldschool racks, that hold the nut in place for the hex-head bolt. Why these were considered desirable is beyond me."
  • "The slimline DS3 patch panels. God help you should you need to do something with the two innermost wires on the back end of that -- there's barely room for pliers, much less fingers."
  • "Any device whose physical characteristics make it a likely candidate to be shelf-mounted, yet which has side ventilation ports which will be blocked by the sides of a rack shelf."
  • "Routers that will accomodate high density of OCx ports but only have the bus capacity to support a fraction of hem."

Vendors, take heed. Messages like the above insert a healthy dose of reality. Have a thought for the poor fellow who has to confront this gear in the middle of the night.

— Mary Jander, Senior Editor, Light Reading

vrparente 12/4/2012 | 11:23:48 PM
re: Network Operators List Their Peeves St,

Interactive services are clearly time of day specific as the systems rotate through servicing folks in various timezones. For example, primetime online service for residential tv, email, reading news, etc. matches local time of day.

So this is the big driver for residential/home web services, streaming video, etc.

So I disagree strongly. Time of day muxing is a very good idea. And my point was that it is easier to do with packet networks -- especially geographically open ones -- like the Internet.

st0 12/4/2012 | 11:23:49 PM
re: Network Operators List Their Peeves vrparente,
like time of day "muxing." is not really a good idea. Many things transmitted at night (low peak time). backup update, consumer goods, banking and even education using the low peak time to transmit the pre-recorded classes. for example, it is really peeves when direct trading bank tells you their computer upgrade everyday between 5-7 am and you can not place a trade....just back on line when you step out of door heading to work...(24 hr trande is really a 22 hr one... )...

The off peak time is more important on speed than daytime...just think about if the bank can shorten their upgrade to one hour.... ha, I might be a millioniar now...May be not!

vrparente 12/4/2012 | 11:23:58 PM
re: Network Operators List Their Peeves Not to comment on RJ45's specifically, but for anyone with operational experience (and more important responsibilities) this is no laughing matter.

Managing cable systems has become a massively complex problem. In the data centers I have worked in (and later the ones I designed with my colleagues) we put significant effort and finances into the management of overhead, underfloor, rack, and equipment cable management. With the increasing density of equipment, sensitivity (meant in BOTH ways) of fiber, and growing size of data centers, it is costly issue.

I've dealt with everything from connectors falling out (call it the gravity effect because some equipment designers love to point the conenctors down/aka south); to issues of alignment, connector types, etc.

In some of the largest data centers we ran we dealt with tens of thousands of RG cables, tens to hundreds of thousands of CAT5/110 pannel connections (that by the way is a tiny operation), hundreds of thousands of CAT5/RJ45, and hundreds of thousands of MMF and SMF tips. And of course a variety of connectors (ST, SC, MT-RJ, MIC, etc....). If that weren't enough managing test equipment for this gets complex as various kinds of test equipment have or don't have appropriate leads/tails. I could go on endlessly. Ribbon connectors, bend radius issues, etc.

In summary a big problem. I hope that wireless technologies will begin to provide some relief. In particular I see wireless taking on a role with out of band network management to replace endless quantities of low-tech crash carts, terminal servers, and fix that serial/tty port pin-out challenge. But wireless has a long way to go to scale from basic p-to-p and p-to-multi-p to supporting large scale hiearchical networks (aka data centers and transport networks). Still though -- how long will it be before folks try to use WLANs to bypass interconnection fees and politics in co-lo facilities ? I'm sure its been done already. One problem replaced by another ...

vrparente 12/4/2012 | 11:23:59 PM
re: Network Operators List Their Peeves I kind of agree with justapeon. But I will add that one can build a packet switched (in this case referred to as a stat muxed) network and get cost savings from opex alone, without accounting for any differences in capital equipment cost or that portion of the opex tied to transmission capacity. In other words, packet switched networks can be easier to operate even for fixed bandwidth.

In reality it is much more complex than that. So the kinds of "stat muxing" we can and do see/use on IP networks, for example, do include things like time of day "muxing." An example is the opreation of a web service for global customers run out of a single geography. So the total transmission capacity (while fixed) is used along different paths at different times of day as people wake up, go to work, come home, go to sleep, etc. In a case like that one could engineer to the max. capacity for a given time of day and traffic load, but then use "stat" geographical multiplexing to save cost, in a way that would be difficult to configure with TDM networks. (Although arguably just as easy with other packet technologies like FR, etc.).

BlueFox 12/4/2012 | 11:24:00 PM
re: Network Operators List Their Peeves Rubber booties on RJ45's as a problem????

I hate those damn things. If you work in a lab and are constantly rewiring your network they are an evil creation. Especially, if you have cables in the slots beside, above, and below the cable you are attempting to remove. Give me the wire cutters and off with the booties!!
mcasaes 12/4/2012 | 11:24:03 PM
re: Network Operators List Their Peeves No, you are wrong, the master/slave your refer is for CPU only, not for switch fabric/forwarding (if you are doing PCAP, the 2nd CPU will even be used to packet capture without impact in the control plane, if doing HA, the 1st and 2nd CPU will be in sync for control plane info)
The current hardware/software in the 8600 is wirespeed for 64 Giga interfaces, no doubts about that. Has been since day one. No "upgrade" was required for that.
Both TAP are active in a dual CPU/SSF configuration.
AFAIK, the full duplex argument come from what is called today Cisco Math..
je 12/4/2012 | 11:24:04 PM
re: Network Operators List Their Peeves A few notes on the PP8600

Each 8691 supports 32g (32g in/32g out) the 128 number is with two 8691's in place

When two 8691 are in place they are both active

The only way the 48 port 10/100 blade is blocking is if there is only one 8691

The Tap Mux is divided in half each side is capable of 4g (4g in/4g out) total 8g for both sides or if you use fuzzy logic 16g (never seen this on their marketechture

Most of their current cards are NON BLOCKING
justapeon 12/4/2012 | 11:24:05 PM
re: Network Operators List Their Peeves >How much oversubscription is acceptable? >How 'bout asking the question: How little >oversubscription is economically viable? Try >creating a data network with no oversubscription >and tell me what it looks like. Then we can have >a more meaningful discussion regarding what you >mean by oversubscription (e.g. does it include >TCP rate adaption?) and how to build a new >blocking/latency model.

In it's most basic form doesn't this really boil down to an architectural question of whether the network in question is built from stat muxes or tdm muxes?

By their very nature a tdm model cannot be over subscribed wherease a statistical model should be (otherwise why not build TDM?)
PO 12/4/2012 | 11:24:10 PM
re: Network Operators List Their Peeves How much oversubscription is acceptable? How 'bout asking the question: How little oversubscription is economically viable? Try creating a data network with no oversubscription and tell me what it looks like. Then we can have a more meaningful discussion regarding what you mean by oversubscription (e.g. does it include TCP rate adaption?) and how to build a new blocking/latency model.

As for vendors using the "full duplex" numbers: this was done by at least some vendors only reluctantly, after certain customers expressed confusion why "other" vendors were citing higher numbers. And for data products, since routes are asymmetric there is at least some technical justification for it.

Then, since data traffic is bursty, and the distribution patterns are not easily generalized, the accurate models are indeed complex. Simplified models are often available, but you're pretty much guaranteed to be operating out of the model at least part of the time.

If customers can suggest improvements, I know plenty of developers will listen with interest.
st0 12/4/2012 | 11:24:11 PM
re: Network Operators List Their Peeves "so how much oversubscription is acceptable? 2/1, 3/1, 4/1? will customers always have to do their own math to get the real story?"
I thought there are "testbed" at North America specifically build to test that kind of stuff and check for competibility.... Are you telling me you have to find it out in the field deployment? (not talk about up grade here, which is totally different story)...

dano4677 12/4/2012 | 11:24:11 PM
re: Network Operators List Their Peeves the 8691 only supports 32g (the 64g number is full-duplex, for whatever it is worth, all vendors use the 'full duplex' argument), and the redundant SFM is only master/slave config.

you cannot get more than 32g out of the current box...nortel has some pretty interesting 'marketechture'...even their 48 port 10/100 cards can be blocking at some point with current backplane and I/O tap.....sure, they say the current platform has a 256g switching capacity, but it is master/slave, and full duplex...break it down and you only get 32g...further, the current tap (tmux) they use for I/O only gets 4g, so again you are limited...it is all smoke and mirrors...the switching capacity on the new 8692 is reported as 512g, but each I/O only supports around 15g (nortel says 19)..and you have to break down the master/slave and full duplex..

the current backplane interface (nortel refers to this as their tmux) only supports 4g also (4 1g channels from each tap), and it is the same for all of their current cards....the interface will change on all of the next gen cards..you can use 8691 series cards with the new SFM (8692), but you will still only be able to get 4g off of a card, since the tmux is limited..so yes, most of their current cards in the 8691 architecture are blocking..

so how much oversubscription is acceptable? 2/1, 3/1, 4/1? will customers always have to do their own math to get the real story?
mcasaes 12/4/2012 | 11:24:12 PM
re: Network Operators List Their Peeves 8690/8691, with 2 of these (as most of the time is the case in a redundant deployment), you have 8Gbps per slot, so the 10G current card has a 10:8 oversubscription.
Regarding the others cards "most of their cards are blocking ?", most of them are non-blocking, the only one that is blocking other than the 10G is the 16p Giga, and this is covered in all documentation.
Other vendors have some magic number math exercise to do, that you just have to be have a PhD to undertand, before knowing what is the actual forwarding rates/features/software combinations you can use at any given time.
dano4677 12/4/2012 | 11:24:13 PM
re: Network Operators List Their Peeves contaminating the 'fiber tip' on an rj45..?

the oversubscription referenced in the last point is neverending...recently spoke with nortel on the 8692 switch fabric for the pp8600...these guys are always fun...

they are offering a 10g module on the current 8691 architecture, but is only capable of 4g to the backplane...8692 goes to 15g per linecard....which cures, but then they release a 30 port gig card? most of their cards are blocking at some point..they put a lot of faith in oversubscription, like everyone else i suppose.....but 100%?
st0 12/4/2012 | 11:24:14 PM
re: Network Operators List Their Peeves Rubber booties normally contain release agent (from the mould) that contaminate the fiber tip and resulted high insertion loss... It is not the case of please someone, but impact performance... (of course, it is someone-else' problem... not the equipment makers...although it is a conatmination created by the equipment maker that implemented the rubber and specified quality...)... typical case study... no body want talk about it... (don't know who complain about it to lightreading... must be a major problem in the field...sorry about the field guys.... if you eye-on the ground, it may discover the rubber booties...).

by the way, the mould release is silicone, not something easy to remove, particularly, after oxidation....(don't tell me it is intend to be a "index matching gel" there).

Eye-in-the-Sky 12/4/2012 | 11:24:15 PM
re: Network Operators List Their Peeves You can please some of the people all of the time, and all of the people some time, but you can't please all of the people all of the time...
Rubber booties on RJ45's as a problem????
st0 12/4/2012 | 11:24:16 PM
re: Network Operators List Their Peeves at current OEM and just in time marketing, design, MFG environment, it is not a news at all:
(1) the decision maker (the guy to select and buy the equipment) is not the one has to wire things up or repair it. (how many eval sheet indicate easy to maintain as requirement?), except VZ's CEO... assume he know something about it...
(2) the designer possibly never "see" the actual CO besides GR requirements... it all brought out using standard box supply anyhow (cheap and time constrain of project... when you done the design, the project time got "eaten" by the delay of state of the art components, etc.etc. the box is the last one on the list)
(3) The CO is normally too small for (a) power, (b) foot print (c) EMI consideration (d) insufficient air flow.... when you try to fit the new gear into the old CO, you meet all the GR but no room for anyone to stand (don't mention of walk... good to get slim size guy and forbid him eat MacDonald.
(4) everything is computer model, the weight does not really matter in the model (how many thermal modeling including the weight calculation?)
(5) etc.etc. don't get me started...

Unless the decision maker (either the guy buy the stuff, or the one make and design the stuff) is the one working in the field for 6 month hands on (you now scare all your MIT/Stanford graduate school kid = state of art designer away), the problem will never fix.... May be the field repair guy should start running for CEO at big place?

Sign In