Heather Kirksey has done both, as a key player in the TR-69 standard and now leader of OPNFV, and offers her upclose view of the contrasts.

July 20, 2015

10 Min Read
An Intimate View: Standards vs. Open Source

How different is open source from the traditional telecom standards process?

One person with intimate knowledge of those key differences is Heather Kirksey, director of NFV for the Open Platform for NFV Project Inc. , the Linux Foundation -backed open source effort. As someone directly involved in developing a recent and enduring telecom standard, TR-69, Kirksey has seen firsthand how both processes work and knows why open source is faster, as the result of a different kind of cooperation.

From her perspective, open source development speeds things up because it helps avoid company politics in reaching decisions and is specifically well-suited to a telecom world in which functionality moves out of hardware into software.

Read more about NFV strategies and the role of open source in our NFV section
here on Light Reading.

In comparing Kirksey's two standards experiences, a short history lesson is in order. TR-69 was developed shortly after the turn of the century and continually updated through the 2000s to solve a critical problem for broadband ISPs: the need to manage and monitor in-home devices including modems and gateways. This is a capability we now take for granted, but in the early days of DSL and cable modems, broadband ISPs were plagued by the inability to quickly discover what was causing a customer problem. Often the broadband line itself was fine, but there was an issue with the modem, or its connection to a PC, or the computer itself that couldn't be remotely detected.

Figure 1: OPNFV Director Heather Kirksey

The result was a massive customer service headache that initially slowed broadband adoption and drove up deployment costs significantly. With each customer added, there were time-consuming and costly issues around installation, maintenance and upgrades for CPE that led to thousands of hours of customer service calls and wasted truck rolls and many angry and frustrated customers.

This process was also killing broadband profits by driving costs up dramatically.

"The process was so manual and had to be so customized to all the different firmware versions and makes and models of devices -- it was an exponentially difficult problem," recalls Kirksey, who initially managed the CPE partner program at Broadjump, which was acquired by Motive, which was later acquired by Alcatel-Lucent (NYSE: ALU). (See Motive, BroadJump to Merge and AlcaLu Gets Motivated.)

Bleeding edge action
As someone managing CPE partnerships, Kirksey found herself at the bleeding edge of solving the issue, but she also discovered her company's service provider customers -- once-familiar names such as BellSouth and SBC Communications - weren't interested in a single company's solution to their problems but were pushing all their vendors to the Broadband Forum -- which might have still been the DSL Forum at that time -- to work out an industry solution.

If that sounds familiar, it's because it is. OPNFV is also an organization driven by network operators who need help with network management and orchestration of their network functions virtualization and want to push the NFV specs forward using open source.

Next page: Progress and pain for TR-69 group

So now we're down to brass tacks: The comparison of what the Broadband Forum did to create TR-69 and what OPNFV is doing with NFV infrastructure.

For a telecom standard, TR-69 actually came together fairly quickly, most likely because it addressed a specific and universal pain point for broadband ISPs -- it's still widely used today, by the way. The group that assembled within the forum to address the issue -- which started out as the auto-config working group, says Kirksey -- spent 18 months arguing over how to accomplish what everyone agreed needed to be done. But after lengthy discussions, including some meetings that ended in the bar, where beer helped fuel the collaborative spirit, the group had -- on paper -- the specifications for how TR-69 could work to enable remote diagnostics and management of CPE.

But that's when things got interesting, Kirksey notes -- and by interesting, she means a bit contentious. The paper spec was given to the engineers at each of the companies involved in the spec at the time, and there were many. Two things happened after that: Either the engineers found the specs incomplete or different people at different companies interpreted them differently and built differing solutions on the same specs.

So the first 18 months of hard work was followed by another 18 months of painful work because implementations of the spec had started but they weren't interoperable, which was a baseline requirement of the broadband operators.

"We had to do a lot in the weeds, a deep dive, painful technical work to get to that next appropriate level," Kirksey says. "People who had invested engineering resources in doing things one way had to go home and tell their engineers 'You have to redo it.' "

Plugfests to the rescue
What ultimately produced the spec that works well today was a series of plugfests at the University of New Hampshire InterOperability Laboratory (IOL) which produced results -- kept confidential at the time so vendors would continue to share information -- that were fed into the Broadband Forum working group, until "every single point of dissent or lack of interoperability" was addressed, says Kirksey, who was co-chair of the group at the time.

She knows this because, along with 2Wire Cofounder Jeff Bernstein, Kirksey created a spreadsheet that tracked every bug, every interop-related issue and every plugfest result. "We made sure each one was addressed and each bug and interop-related issue was fixed," she recalls.

Relationships became a key part of the process -- figuring out who was intent on getting work done and who was a political grandstander, and understanding when to defer to a participant because of their specific expertise, were both essential factors, she says. Kirksey credits TR-69's staying power to the group's determination to work through even the "big ticket items" to deliver true interoperability.

Next page: How it works in open source

Fast forward to today, and Kirksey credits that experience and what she learned with qualifying her for her current job at OPNFV -- and a process that is very different.

One overarching difference is that the open source process builds on code, not specs drawn up in advance of implementation, and that shortens the process considerably.

"It also helps circumvent some of the political aspects of the fighting," Kirksey comments. "Because you can very easily argue this way is better than that way and to have long arguments to back it up. But once you've got cold hard code, that can put the kibosh on a purely political argument."

That doesn't mean conceptual discussions go away entirely, she adds, as some are still necessary, "but this process reduces those to the necessary ones instead of the grandstanding ones."

The collaborative process itself changes somewhat -- at least that has been Kirksey's experience -- because tough technical problems are now things that the group has to work together on in a different way. "When you are beginning to write code together, that actual activity and process encourages more collaboration," she says. "Because you are rolling up your sleeves together to solve the problem and that naturally just leads to more collegial happenstance than if you have [everyone around] the table doing something more like contract negotiations."

There is also respect given within the group to the arguments of anyone willing to go and create code to back up what they are saying. "To call it 'put up or shut up' sounds unnecessarily harsh," Kirksey says. "It's more like, if you are passionate about something enough to go and create code, that is going to be considered by the community, a lot more than throwing sticks from the sidelines. There is a certain respect for people who put the work in and that helps resolve issues."

Agile principles key
Two other key differences were apparent in OPNFV's process of developing its first release, Arno, which came out earlier this summer. First, open source uses agile principles, as opposed to a more traditional waterfall approach and so it can actually seem more chaotic, especially at first.

"There is less of that top-down, upfront decision-making, so it can feel a bit more chaotic," Kirksey says. "You can perhaps ship a spec out the door on an arbitrary date but if your builds aren't working, you aren't going to ship a release. Working through the bugs to get the release out can feel more chaotic. But that's a good thing because it allows us to not pick winners."

And that's the second thing that's different about open source -- not picking winners and losers in the standards process.

"I'll use an example from OPNFV: When we were first talking about Arno, lots of people were saying we should use installer X," she says. "In a traditional standards world, we would have had some period of time where we would have argued the merits of installer X versus Y versus Z for months until we reached the point where we said, 'Okay we are going with installer Y and all of our decisions would have flowed from that.' "

Instead, OPNFV established requirements for any installer that is going to be part of the Arno release -- its first -- and at the point of release, two installers were ready and met those requirements. Kirksey expects more will be coming, but OPNFV doesn't try to arbitrate or pick a winner.

OPNFV was six weeks late with that first release, missing the regular cadence of a release every six months, mostly due to the enormity of its initial task, Kirksey says. In telecom terms, that wasn't a significant lag, and was still a much faster and more streamlined process than what would have happened if each service provider worked through these issues separately with its vendors.

"We were integrating all the components together in a way that no one had done before, in an automated repeatable way, and we were finding bugs, configuration issues, people having different assumptions, solutions that worked on one type of hardware and not another when we wanted it to be hardware-agnostic," she says. "Those issues all took a while to solve."

Because the community is working together on the code, it is also easier to catch bugs or other issues, Kirksey notes. "There's a saying: given enough eyeballs, every bug is shallow. That is where the speed comes from." And the collective work of the community replaces many more man-hours that would have been spent in individual labs, making this a more efficient process even if it seems more chaotic at times.

Release cadence critical
The shift to a software-driven universe also tilts the scale toward the open source approach, which uses iterations and a cadence of releases to constantly update and improve, rather than relying on purpose-built hardware with substantially less flexibility.

"When you are doing something like a protocol and you know people are going to go off and implement it, and that protocol is going to be in all the hard-to-upgrade legacy hardware on the planet, there is a great burden to make sure you get it exactly right," Kirksey notes. "With open source, where you are updating it at a roughly six-month cadence, you have a bit more leeway to be experimental and then fix problems, if they arise."

Since NFV is specifically designed not to be monolithic, but to be flexible and offer the freedom to experiment with new things and turn services up and down as needed, the iteration process enables the faster change that is appropriate.

And the cadence of the release schedule also imposes a discipline on the process that keeps the focus on timely problem solving. "That's where the bigger change -- the move to agile processes versus waterfall -- really has an impact," Kirksey notes. "Instead of feeling like we have to have everything perfect, we design our processes around being able to iterate and have that freedom."

— Carol Wilson, Editor-at-Large, Light Reading

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like