The IP Priesthood
Do we find ourselves in an analogous situation today in IP networking?
IP has become the epicenter of our industry (if not popular culture). From a protocol-stack point of view, people have the religion. They are now planning to run just about any network service over IP: Ethernet-over-IP, Fibre Channel-over-IP, SCSI-over-IP, Frame Relay-over-IP, TDM-over-IP, ATM-over-IP, maybe even Automobiles-over-IP. With the help of MPLS and an armada of encapsulation and interworking standards, the IP network might potentially deliver almost any network service a carrier might want to offer.
Whether you buy this IP-über-alles vision or not, it is clear that IP technology is strategic to any networking vendor. Which brings us back to the Renaissance. As in the church of that time, the keys to the IP kingdom are held in the hands of an astonishingly small technical community: the "IP Priesthood." These are the 50 or so software engineers who specialize in the routing protocols that determine and communicate network topology – BGP, OSPF, IS-IS, to name a few of the key ones. These engineers are aided and abetted by their counterparts in the service provider world, a similarly elite corps of network engineers who build and operate the world’s major IP backbones.
In 1517, Martin Luther walked up to the castle church in Wittenberg, Germany, nailed his 95 Theses to the door, and the Reformation was on. Will the same thing happen to IP? And why hasn’t it happened already? After all, there is a lot of money at stake. IP routing protocol engineers command premium salary and option packages. Landing a big-name member of the priesthood on your engineering team can be a company-making event. Adam Smith taught us that such demand would inevitably provoke ample supply. But the scarcity persists.
If it were as simple as throwing money and brains at the problem, we’d be swimming in IP software engineers. The IP have-nots have burned billions of dollars trying to acquire IP routing expertise. Witness Nortel Networks Corp.'s (NYSE/Toronto: NT) oft-cancelled router projects and Lucent Technologies Inc.'s (NYSE: LU) abandoned Nexabit acquisition. While universities have been teaching IETF routing specs for years and pumped out a generation of IP-literate PhDs, the state of IP routing expertise remains much as it was many years ago — a small pool of talent controlled by a handful of companies.
One theory is that in the IP domain, real-world experience is vastly more important than university training or corporate development dollars. The practice of building and running large routed networks is quite different than simply implementing the IETF specs. Amar Gupta, founder and CTO of Amber Networks (an edge router company recently acquired by Nokia Corp. [NYSE: NOK]) emphasizes this need for real-life WAN deployment experience, and goes on to highlight the huge gulf between publicly available protocol stacks (the starting point for most startups) and scaleable, highly available software.
And for most of the booming period of the Internet’s existence, there has been only one company in the game: Cisco Systems Inc. (Nasdaq: CSCO). Therefore, if you haven’t been writing router code for Cisco, you just don’t have the knowledge to do a credible implementation (or so this theory goes).
A darker argument focuses on interoperability. This theory (heard mainly from Cisco’s competitors) postulates that Cisco’s own routing software deviates from published standards in many key respects. Since the Internet until recently has been primarily a “Cisco-powered network,” what matters to customers is therefore not IETF compliance, but Cisco compatibility, including the bugs. Conspiracy theorists contend Cisco intentionally inserts knuckleballs into its implementations; others say the company is collecting customer-specific coding to solve real problems. The end result is the same: If you don’t know Cisco’s non-standard work chapter and verse, you are out of luck.
This does not mean that Cisco’s engineers are any smarter than all of the other bright people in networking. It just means they know where the skeletons are buried. Key routing engineers are well aware of this, which is one reason they often stick together as teams. As one elite routing engineer told me in a recent interview for one of Accel’s startups, “Sure I know I can do a good job on BGP. But who is going to do the other protocols?” His trust extended only to those he knew on his team at Cisco.
Just as important is the issue of testing. Many believe that the only way to develop core-capable IP routing software is to have some of the key backbone service providers hammer away at the code, repeatedly, for extended periods. While most any vendor would be happy to participate in such a drill, the small number of key IP engineers in these backbone providers won’t do this for just anyone, as it is a major investment of their own time and a risk for their network. This testing bottleneck is one of the most formidable entry barriers to new router vendors.
It is widely accepted that a key to Juniper Networks Inc.'s (Nasdaq: JNPR) success was the willingness of the IP service provider community to spend significant effort on the pre-release of JunOS software. Legend has it that in the pre-Juniper era, UUNet spent over a year working with the startup NetStar (later acquired by Ascend) on their software, in an effort to bring up a rival to then-monopolist Cisco. NetStar had some fast hardware but lacked the Cisco-specific knowledge needed to succeed, and, despite UUNet’s support, this was ultimately an effort in vain.
Such dogma-busting apparently had to wait until much of the Cisco routing team migrated wholesale over to fledgling Juniper. Finally the industry had an entity with both the knowledge to achieve Cisco “bug compatibility” as well as the service provider goodwill to enable a painstaking testing effort. Even for Juniper it was no mean feat, perhaps because Cisco code itself is a moving target. Between the time of the Cisco team’s exodus and the availability of initial Juniper software, the Cisco code had evolved significantly, with changes that had to be discovered and implemented via the testing process.
The list of other vendors that have been able to pull off this feat is perilously short, leaving our industry in the current predicament of routing talent scarcity. The bottleneck is opening a bit, but slowly. Cisco and a few others are training and then losing software engineers, spreading the DNA a bit more broadly. IP carriers also seem to realize it is to their benefit to support new router vendors via testing. My friend Jim McManus, formerly VP Engineering at UUNet during the glory days, contends that in fact this issue of artificial scarcity is already behind us. What remains are two or three years of hard work, an investment of time and energy that he feels precious few companies have been willing to make.
Somewhere, out there, the Martin Luther of IP routing is toiling away. But for now, the power of the IP priesthood remains a force to be reckoned with.
Peter Wagner is a general partner at Accel Partners, a venture capital firm based in Palo Alto, Calif. He has worked in communications and computing since the mid-1980s as a physicist, line manager, and venture investor. Disclosure note: Accel Partners was a lead investor in both Amber Networks and UUNet during their startup phases.