Optical/IP Networks

The IP Priesthood

Centuries ago, an all-powerful church dominated Renaissance Europe. While a broad community drove scientific and cultural innovation, a firm monopoly was maintained on spiritual life. Galileo might have been able to discern the workings of the solar system, but if he wanted to talk to God he needed to find a priest to translate.

Do we find ourselves in an analogous situation today in IP networking?

IP has become the epicenter of our industry (if not popular culture). From a protocol-stack point of view, people have the religion. They are now planning to run just about any network service over IP: Ethernet-over-IP, Fibre Channel-over-IP, SCSI-over-IP, Frame Relay-over-IP, TDM-over-IP, ATM-over-IP, maybe even Automobiles-over-IP. With the help of MPLS and an armada of encapsulation and interworking standards, the IP network might potentially deliver almost any network service a carrier might want to offer.

Whether you buy this IP-über-alles vision or not, it is clear that IP technology is strategic to any networking vendor. Which brings us back to the Renaissance. As in the church of that time, the keys to the IP kingdom are held in the hands of an astonishingly small technical community: the "IP Priesthood." These are the 50 or so software engineers who specialize in the routing protocols that determine and communicate network topology – BGP, OSPF, IS-IS, to name a few of the key ones. These engineers are aided and abetted by their counterparts in the service provider world, a similarly elite corps of network engineers who build and operate the world’s major IP backbones.

In 1517, Martin Luther walked up to the castle church in Wittenberg, Germany, nailed his 95 Theses to the door, and the Reformation was on. Will the same thing happen to IP? And why hasn’t it happened already? After all, there is a lot of money at stake. IP routing protocol engineers command premium salary and option packages. Landing a big-name member of the priesthood on your engineering team can be a company-making event. Adam Smith taught us that such demand would inevitably provoke ample supply. But the scarcity persists.

If it were as simple as throwing money and brains at the problem, we’d be swimming in IP software engineers. The IP have-nots have burned billions of dollars trying to acquire IP routing expertise. Witness Nortel Networks Corp.'s (NYSE/Toronto: NT) oft-cancelled router projects and Lucent Technologies Inc.'s (NYSE: LU) abandoned Nexabit acquisition. While universities have been teaching IETF routing specs for years and pumped out a generation of IP-literate PhDs, the state of IP routing expertise remains much as it was many years ago — a small pool of talent controlled by a handful of companies.

One theory is that in the IP domain, real-world experience is vastly more important than university training or corporate development dollars. The practice of building and running large routed networks is quite different than simply implementing the IETF specs. Amar Gupta, founder and CTO of Amber Networks (an edge router company recently acquired by Nokia Corp. [NYSE: NOK]) emphasizes this need for real-life WAN deployment experience, and goes on to highlight the huge gulf between publicly available protocol stacks (the starting point for most startups) and scaleable, highly available software.

And for most of the booming period of the Internet’s existence, there has been only one company in the game: Cisco Systems Inc. (Nasdaq: CSCO). Therefore, if you haven’t been writing router code for Cisco, you just don’t have the knowledge to do a credible implementation (or so this theory goes).

A darker argument focuses on interoperability. This theory (heard mainly from Cisco’s competitors) postulates that Cisco’s own routing software deviates from published standards in many key respects. Since the Internet until recently has been primarily a “Cisco-powered network,” what matters to customers is therefore not IETF compliance, but Cisco compatibility, including the bugs. Conspiracy theorists contend Cisco intentionally inserts knuckleballs into its implementations; others say the company is collecting customer-specific coding to solve real problems. The end result is the same: If you don’t know Cisco’s non-standard work chapter and verse, you are out of luck.

This does not mean that Cisco’s engineers are any smarter than all of the other bright people in networking. It just means they know where the skeletons are buried. Key routing engineers are well aware of this, which is one reason they often stick together as teams. As one elite routing engineer told me in a recent interview for one of Accel’s startups, “Sure I know I can do a good job on BGP. But who is going to do the other protocols?” His trust extended only to those he knew on his team at Cisco.

Just as important is the issue of testing. Many believe that the only way to develop core-capable IP routing software is to have some of the key backbone service providers hammer away at the code, repeatedly, for extended periods. While most any vendor would be happy to participate in such a drill, the small number of key IP engineers in these backbone providers won’t do this for just anyone, as it is a major investment of their own time and a risk for their network. This testing bottleneck is one of the most formidable entry barriers to new router vendors.

It is widely accepted that a key to Juniper Networks Inc.'s (Nasdaq: JNPR) success was the willingness of the IP service provider community to spend significant effort on the pre-release of JunOS software. Legend has it that in the pre-Juniper era, UUNet spent over a year working with the startup NetStar (later acquired by Ascend) on their software, in an effort to bring up a rival to then-monopolist Cisco. NetStar had some fast hardware but lacked the Cisco-specific knowledge needed to succeed, and, despite UUNet’s support, this was ultimately an effort in vain.

Such dogma-busting apparently had to wait until much of the Cisco routing team migrated wholesale over to fledgling Juniper. Finally the industry had an entity with both the knowledge to achieve Cisco “bug compatibility” as well as the service provider goodwill to enable a painstaking testing effort. Even for Juniper it was no mean feat, perhaps because Cisco code itself is a moving target. Between the time of the Cisco team’s exodus and the availability of initial Juniper software, the Cisco code had evolved significantly, with changes that had to be discovered and implemented via the testing process.

The list of other vendors that have been able to pull off this feat is perilously short, leaving our industry in the current predicament of routing talent scarcity. The bottleneck is opening a bit, but slowly. Cisco and a few others are training and then losing software engineers, spreading the DNA a bit more broadly. IP carriers also seem to realize it is to their benefit to support new router vendors via testing. My friend Jim McManus, formerly VP Engineering at UUNet during the glory days, contends that in fact this issue of artificial scarcity is already behind us. What remains are two or three years of hard work, an investment of time and energy that he feels precious few companies have been willing to make.

Somewhere, out there, the Martin Luther of IP routing is toiling away. But for now, the power of the IP priesthood remains a force to be reckoned with.

Peter Wagner is a general partner at Accel Partners, a venture capital firm based in Palo Alto, Calif. He has worked in communications and computing since the mid-1980s as a physicist, line manager, and venture investor. Disclosure note: Accel Partners was a lead investor in both Amber Networks and UUNet during their startup phases.
Page 1 / 7   >   >>
laserbrain 12/4/2012 | 7:44:22 PM
re: The IP Priesthood more like the IP big-leagues. no matter how big the salaries, there's still a shortage of the great ones.

maybe if there were brain steroids as strong as the juice Barry Bonds is on, we'd have a surge in router jocks like we do home run hitters. Would you sacrifice your liver and various, um, important organs for Tony Li power?
beowulf888 12/4/2012 | 7:44:15 PM
re: The IP Priesthood Oh, give me a break! Conspiracy? Are there are thousands upon thousands of top-rate SONET engineers out there? And ATM engineers? The only reason there's dearth of top-rate routing protocol coders, is that TCP/IP has spread like wildfire with the growth of the Internet. The demand has far outpaced the supply. I like Lightreading, but it's sort of a cross between an industry publication and a supermarket tabloid. Well, Elvis is alive and he's a BGP guru...

skeptic 12/4/2012 | 7:44:13 PM
re: The IP Priesthood
The reason for things being the way they are
is that protocol specifications for BGP et al
are next to useless. Anyone who tries to use
them as a guide to develop a router will fail

Why is this the case? Because the IETF as
a standards organization is a fraud. There
is an IETF for the public in terms of the
meetings and the mailing lists, but none of
the real decisions are made there. The
real decisions are made in hotel suites or
in private email discussions.

Why are the standards for BGP and certain other
protocols so bad? You will get three answers

1. I dont know you, so you are obviously wasting
my valuable time.

2. We have to be "vendor netural" (therefore
promoting an real-world interoprable BGP
standard is somehow seen as favoring cisco
- of course the gated people might actually
still believe this I guess)

3. This isn't an important issue for the IETF.
(they are working the serious issues like
which working group gets TDM over MPLS).

As far as cisco goes, IOS is a mess. Even
cisco doesn't have as much knowledge about
it as people would think. And many of the
people selling themselves as "experts" don't
really know all that many "secrets".

There is plenty of so-called router "talent"
with a miserable track-record of failed startups
behind them. The worst are nothing better
than leeches. There are VCs who are so ignorant
that they think that serving as the chair of
a working group translates into practical
product development skills. Most of the time
it doesn't.

Its not just a question of talent. You have
to hire people who can:

1. Work in a team.

2. Share information with co-workers rather
than playing "I've got a secret".

3. Be willing to write code (rather than
sit on the phone all day and talk to friends).

4. Be able to treat other engineers as
equals. (rather than hanging around the execs
and the marketing people all day).

5. Have some track-record in delivering successful
products. Just because you maintained IOS does
not mean that you can write new routing protocol
software. They are very different skill-sets

6. Are committed to the success of the company.

These seem like simple things, but they are

As far as the people who claim to have all
the secrets, most of them don't. Their ideal
interview is to sit in front of some manager
and play all smart. If you start asking them
the specifics of the "secrets" of IOS that
they know, they will either say nothing
or just collapse.

The example of netstar (in the original article)
is hilarious. Netstar failed not because of
cisco secrets, but because their testing effort
was a joke and they could never get gated to
run stable. They had fundemental problems
like software crashes and inconsistant forwarding
tables. Thats not cisco's fault and that
would not have been fixed by hiring any amount
of high-priced experts.

beowulf888 12/4/2012 | 7:44:10 PM
re: The IP Priesthood IETF standards are free. There's something to be said for not having to dole out beaucoup bucks to read something that will put you to sleep.

beowulf888 12/4/2012 | 7:44:10 PM
re: The IP Priesthood Skeptik:
Sounds like somebody needs a cookie and nap ;-).

Standards documents always leave something to be desired, but the IETF has a much better track record for developing implementable protocols than just about any other standards body.

I would highly recommend a book entitled "Open Systems Networking: TCP/IP and OSI" by Dave Piscitello. It's a little out of date, but he examines the standards development process of both those camps, warts and all.

Yes, I would agree with you that the within the IETF there are many decisions "made in hotel suites or in private email discussions". But the characteristic that has made the IETF successful is that there has always been a "make it work" and then "finish the standard based on how we made it work" mentality. With other standards organizations, such as industry groups or government-mandated groups, there seems to be "make a standard" and then "lets make it work to conform to the standard" mentality. The former's cultural perspective no doubt has its roots in the hacker ethic that came out of the universities, while the later seems more beholden to the bureaucratic ethic. Why else has TCP/IP been so successful?

BTW: When is comes to obfuscation, who can beat the ITU?

skeptic 12/4/2012 | 7:44:02 PM
re: The IP Priesthood Standards documents always leave something to be desired, but the IETF has a much better track record for developing implementable protocols than just about any other standards body.

Lets put the historical propoganda aside. What
the IETF may have been years ago, it isn't today.

When the whole organization starts to function
in secret, it ceases to be an organzation where
people can get anything useful from or make
any contribution to. I mean it was one thing
for some of the higher level parts of the IETF
to operate in secret, but what sense does it
make to run working groups that way?

Its a to a point where more information about
what the IETF is doing can be found out from
watching cisco than it can from following the
mailing lists.


I would highly recommend a book entitled "Open Systems Networking: TCP/IP and OSI" by Dave Piscitello. It's a little out of date, but he examines the standards development process of both those camps, warts and all.


I lived through parts of that. I don't need
to read about it and lord knows I would never
recommend what the ISO did in standards as a
good thing.


Yes, I would agree with you that the within the IETF there are many decisions "made in hotel suites or in private email discussions". But the characteristic that has made the IETF successful is that there has always been a "make it work" and then "finish the standard based on how we made it work" mentality.

I don't agree with you on that. And would point
to what happened in diffserv as a classic
example of people who were not interested
in making it work or finishing the standard.

The "senior" people in the IETF, who are so
fearless about tearing people they don't know
apart, lacked the guts (or the interest) to
say a word against the great Van Jacobson no
matter how much damage he did or how many times
he would play "I know best because I know more
than you but I can't tell you what I know".
He may or may not be right, but its impossible
for people to productively work on standards
with this sort of thing going on.

And in the end, the answer always seems to be
to let the group fold and do something else
rather than fix whats wrong. There are people
fighting whiny battles in public at IETF over
nothing who will not step in and fix things
that are clearly off-track or wrong.

If you want another example, look at the so-called
"policy" efforts. Based on the bad ideas of
a few influential people, the whole thing has
been knocked off the rails and re-directed
away from solving useful problems of real people
into something that few (if any) people are
going to want. (whatever year they finish).

I certainly don't mean to say the ITU is a
good approach. What I want is for the IETF
to operate like its theorically supposed to.

netskeptic 12/4/2012 | 7:43:59 PM
re: The IP Priesthood > Lets put the historical propoganda aside. What
> the IETF may have been years ago, it isn't
> today.

Yes, and the change did happen a long time ago.
At the same time other standard organizations are even worse (e.g. ATM Forum).

So, I suspect that it is going to be a reformation indeed.



nonobvious 12/4/2012 | 7:43:56 PM
re: The IP Priesthood I think at this point most people realize that the IETF has degenerated into a forum for industry deadbeats to spew endless smart-sounding drivel in support of inflated option packages, resumes, and egos.
Bumper_car 12/4/2012 | 7:43:36 PM
re: The IP Priesthood Designing and building supportable routed networks has for years been and remains more of an art form than a science. The complexities of application/user usage and loading while trying to provide some level of system/circuit failure protection remains too complex for computers to model properly, particularly for large networks. Many of inherent characteristics of the interworkiing between the complex "circuits", the Ethernet LANs, and the routing protocols are undocumented and have to be infered by people with many years of experience in building large networks.

There is a saying about the definition of a data networking professional: Some one that has been around long enough to have "screwed up" and seen enough "screw ups" to know what not to do. The problem with most of the younger people (including those at startups) is that they do not know what to avoid. There are some briliant people out there that are trying to design and code network protocols, they just don't have the experience.

Having a doctorate in data communications, math or some other "science" can and never will substitue for experience. More than anything else, what the IETF is currently suffering from is a lack of experience. They don't know what not to do.
tony1athome 12/4/2012 | 7:43:32 PM
re: The IP Priesthood Would you sacrifice your liver and various, um, important organs for Tony Li power?

Trust me, it's not worth it.

Page 1 / 7   >   >>
Sign In