NFV Tests & Trials

Testing Times for Our Industry

Independent testing of network hardware used to be a big deal. In fact, back in the 90s, when I was an editor at Data Communications Magazine, we had a full-time test editor who did nothing but commission independent tests, and they were always the most contentious and widely talked about features we published.

There was a good reason for that (other than the tests were first rate). At the time, disruptive networking technologies, such as Ethernet switching, were making their way onto enterprise networks, and network managers were desperate for authoritative, quantitative test data on how well the products from different companies performed and interworked.

Today, the telecom industry is faced with a slew of new and violently disruptive New IP technologies -- virtualization, open source, white box, IoT, 5G, telco cloud and on and on -- that in terms of complexity make Ethernet switches look like child's play.

In the face of all of this technology, you would think that the Internet would be bursting at the seams with independently published tests of these new communications solutions. But it isn't. In fact, at exactly the time when network architects' need for independent testing has never been greater, in the first ten years of the 21st century it simply disappeared (replaced, largely, by increasing amounts of marketing spume and hot air from vendors).


The answer is tied to the fate of the business-to-business (B2B) tech publishing industry (a matter close to my heart, having alternated between a victim and a beneficiary of its downfall).

I'm With Stupid
With the arrival of the Internet at the end of the 20th century, a historic "wave of stupid" swept across B2B publishing.

All of the magazine publishers took one look at "this Internet thing" and had a massive panic attack, throwing all of the rules that had kept their businesses ticking over for 100 or so years out of the window -- little things such as editorial integrity, paying professional writers a living wage, fact checking stories and generally giving a toss about producing quality content.

One of the first editorial features to be cut was independent laboratory tests of new technology.

Well, the Internet turned out to be the magic bullet that passed straight through those publishers' heads (missing their brains by about six feet) and killing all of the old tech publishing brands stone dead. Some of them are still stumbling along, Walking Dead style, but none of them successfully made the transition to the Internet age.

The story has a happy ending though, particularly if you are a network architect at a service provider that's looking for independent test results of technology being developed by the vendors focused on New IP developments such as virtualization.

Because this year, three remarkable test-related events followed one after another.

First, Cisco Systems Inc. (Nasdaq: CSCO) asked us if we would work with our long-term test partner, European Advanced Networking Test Center AG (EANTC) , to perform and publish an independent Light Reading test of its NFV and cloud solutions. That test, the results of which were published here, was itself a huge breakthrough and had a massive impact in an industry looking for evidence that virtualization was real. (See Validating Cisco's Service Provider Virtualization & Cloud Portfolio.)

Then, we asked if we could run another test, with more detail: Cisco, again, said yes.

You can read the results of our extensive second Cisco virtualization evaluation right now on Light Reading. (See Validating Cisco's NFV Infrastructure Pt. 1 and Validating Cisco's NFV Infrastructure Pt. 2.)

This time out we collaborated with EANTC to evaluate Cisco's ability to deliver the four essential functions that service providers must have to make the move to 21st century cloud-based services: performance; reliability; multi-service capabilities; and so-called "single pane of glass" management.

My analysis of the results is that Cisco has again raised the bar for the comms industry. In fact, while the NFV market overall is entering something of a trough of disillusionment, Cisco's test results show it starting to head in the opposite direction, pulling away from many of its competitors.

The second thing that happened is that other vendors started to ask Light Reading to test their products, something that, with the exception of Cisco, hadn't happened for a dozen years.

I'm happy to reveal that Light Reading's test partner, EANTC, is about to start work on an historic test of Alcatel-Lucent (NYSE: ALU)'s virtualized routing capabilities. Expect to read the results in January 2016.

And we're also preparing to evaluate another world-leading IP company’s virtualization solutions in Q1 next year (we'll be announcing the identity of that company soon, I hope).

It's all incredibly exciting and also really great news for the industry; it's exactly the information service providers need, when they need it.

The third test breakthrough this year came when we issued invitations to every virtualization vendor in the industry to take part in the first ever public NFV interoperability test and a whole slew of companies said "Yes" (including all four of the top IP manufacturers by market share).

I was expecting three or four participants for this interoperability evaluation, which is being conducted in anticipation of the launch in 2016 of the New IP Agency (NIA), a not for profit which will be dedicated to providing education and testing around NFV. But when the number of participants hit 13, we had to close admissions (a nice problem to have). (See Good News for NFV Interoperability.)

For the record, the 13 companies signed up now are: Alcatel-Lucent; Cisco; Cobham; Hitachi; Huawei; IneoQuest; Juniper; Metaswitch; NetNumber; Netrounds; Overture; Procera; and Sonus.

Cranking up the testing dial
The bottom line here is that testing is back, big time, at Light Reading, and we are going to be amping up our test program in 2016.

As ever, Light Reading will pay for all our tests, to ensure that they are really independent and valid. (If Light Reading commissions a test and the product or technology doesn't work, we'll tell you that -- it's what we do -- and the companies that submit their products to be tested by us know that... that's why our tests matter.)

In addition to the LR vendor-specific tests, like the ones undertaken with Cisco and, now, Alcatel-Lucent, we will also be working with the soon-to-be-launched New IP Agency to create Phase 2 of interoperability testing, around the critical area of orchestration.

I couldn't be happier that independent testing is making a resurgence and that Light Reading and the NIA are at the heart of it. And I salute the industry vendors that have created this comeback by pushing us to test their products.

Kudos, comms industry!

— Stephen Saunders, Founder and CEO, Light Reading

kq4ym 11/17/2015 | 2:56:46 PM
Re: did you miss this one? "Cisco has again raised the bar for the comms industry" and getting back to testing certainly will drive the industry forward in ways that can't be bad. With Cisco leading the charge, others are certainly going to join in, and at the same time improviing product design and implementations.
tojofay 11/9/2015 | 1:02:59 PM
Re: did you miss this one? Lippas


tojofay 11/8/2015 | 5:03:12 PM
Re: did you miss this one? Thanks - point taken. A murky issue for sure. Event sponsorship, advertising in publcations, and all kinds of "support" can taint an "independent" test. I'm curious; are you disputing the results of the testing? Chalenging the integrity of the publication? I'll send your comment to Lippis and see what he thinks. Looking forward to LR testing INFN's latest gear.

baznyankee 11/8/2015 | 4:28:14 PM
Re: did you miss this one? http://www.infinera.com/wp-content/uploads/2015/10/Lippis_Report-Low_Latency_Data_Center_Interconnect-Cloud_Xpress_Arista_Data_Center_Switches.pdf

While I couldn't say who paid for this test, I suspect it's as independent as the Lippis Report is credible (35,000 subscribers lends some cred.).

Steve Saunders 11/8/2015 | 3:50:04 PM
Re: did you miss this one? Tojofay, do you know if that test was really independent? To be independent means more than just adding that word to a custom test. 

If the participants paid the company that did the test it's not indepndent. 

That's not a semantic difference - it really matters. 

I can't tell if the participants in that test paid Lippis to test their equipment or not. 

When LR does an independent test we make sure its CLEAR who paid for it - we did!
mhhf1ve 11/5/2015 | 7:15:03 PM
more independent testing FTW! It's amazing how valuable independent testing can be... journalism isn't dead -- it's just really difficult to do properly.

But then there are some folks out there with a noble mission to make sure products are tested thoroughly!

Ariella 11/5/2015 | 2:28:52 PM
Re: did you miss this one? It's a good thing we don't suffer from triskaidekaphobia around here.
[email protected] 11/5/2015 | 1:03:25 PM
Re: did you miss this one? I just saw that, a few minutes after this article was loaded into our publishing system.

That's another very credible test and all the participants are to be applauded.

But it's not a test of NFV infrastructure... independent testing of the most important and high profile communications networking development (the introduction of software-defined networking and VNFs) is almost non-existant.

Believe me, we will be VERY keen to hear about other such independent tests, undertaken by test firms such as EANTC, and we'll highlight those too. 
tojofay 11/5/2015 | 12:29:12 PM
did you miss this one? http://finance.yahoo.com/news/infinera-arista-validate-performance-low-163455572.html


Sign In