& cplSiteName &

EXCLUSIVE! NFV Interop Evaluation Results

Light Reading
Prime Reading
Light Reading

Of all the industry test results out there, this is the one that people have been waiting for.

In a historic first for the communications networking industry, Light Reading, in partnership with EANTC on behalf of the New IP Agency (NIA), is happy and proud to announce the publication of the world's first independent interoperability evaluation of NFV infrastructure, focused (in Phase 1) on multivendor NFV infrastructure-to-virtual network function (VNF) interoperability.

These results are just a first step on a very important journey for an industry that is in dramatic upheaval. Never before have network operators, faced with the unknown and daunting challenges of introducing virtualized functions and their supporting infrastructure platforms, needed independent test results as much as they need them now: Such independent evaluations can cut down development times, save operators resources, and help speed up further tests, field trials and live deployments.

That's why we have been moving fast. The Phase 1 evaluation, which was publicized, set up and completed in just a few months (an extraordinarily short time for such a procedure), included 12 vendors, which between them submitted four NFV infrastructure (NFVi) platforms and 17 VNFs.

The NFVi vendors, which all offered platforms based on OpenStack, were:

  • Alcatel-Lucent
  • Cisco Systems
  • Huawei Technologies
  • Juniper Networks

The VNF vendors were:

  • Alcatel-Lucent
  • Cisco Systems
  • Cobham Wireless
  • Hitachi Communication Technologies America
  • Huawei Technologies
  • IneoQuest Technologies
  • Juniper Networks
  • Metaswitch Networks
  • NetNumber
  • Netrounds
  • Procera Networks
  • Sonus Networks

The VNFs came in many shapes and forms, from virtual routers, EPC (evolved packet core) and IMS components, firewalls, test probes, deep packet inspection (DPI) modules and session border controllers.

The testing focused on the functional interoperability of NFVi platforms and various VNFs, but did not include the sort of tests more commonly associated with communications infrastructure, such as performance, scalability and resilience testing: In addition, these tests did not include any management and orchestration (MANO) evaluations (they will be the focus of a later evaluation phase).

The tests carried out by the EANTC team, which were all conducted remotely (another new feature!), were of NFVi-VNF interoperability combinations that had never been tried before in such open conditions. As a result, not all were successful, but as this is still a very nascent market, it was expected that some of the evaluations were not be completed successfully.

In fact, the success rate -- 25 of the 39 combinations tested, or 64%, passed -- was "a great result," noted EANTC managing director Carsten Rossenhövel.

That higher-than-expected pass rate was due, in part, to the high degree of industry participation: During the seven weeks of testing, 55 engineers from various vendors (including test system partner Ixia) were involved, with support from 14 marketing and communications executives, seven EANTC staff and three from Light Reading/NIA: In total, 79 people were directly involved in the process.

So does that mean that there was a 36% fail rate? It could be viewed that way, but that would be shortsighted. The point of these evaluations is not to identify technology that 'does not work' – it is to see what can be done better, what needs to be fixed. The NIA exists to advance the industry and speed up the process of getting useful technology to market: When the evaluations highlight an interoperability combination that doesn't work, that means the vendors can fix the problems! This can only be good for the industry.

The evaluation also highlighted many other interesting and useful outcomes, not least of which was insight into the performance of OpenStack. For some OpenStack supporters, it is a viable alternative to standardization: "If only everybody would base their implementations on OpenStack, all interop issues would be solved," could be a phrase you might hear in virtualization circles.

Well, according to EANTC's results, OpenStack is far from ready to play an interop role. "There were tons of interop issues despite the fact that all NFVis were based on OpenStack," noted EANTC's Rossenhövel. That's because there are far too many options (including how to manage a VNF), too many versions (with backwards compatibility an oversight), and items left open for vendor consideration (including license management).

So there are many positives to take away from the Phase 1 test results, but there is much, much more to be done: A test combination that passed confirms that the initial lifecycle management tasks work between the parties on a functional basis, but that still leaves many aspects of application-layer functionality, performance, resiliency and orchestration to be evaluated. Those aspects will be the focus of future evaluations.

In the meantime, the following pages will provide you will a full matrix of the test results and detailed insights into the evaluations, the outcomes and the conclusions. So dive in!

Page 2: The Participants

Page 3: Test Setup and Coverage

Page 4: The Results Matrix

Page 5: Interoperability Findings and Challenges

Page 6: Further Interoperability Findings and Challenges

  • Prefer to read this report in a downloadable PDF file? An extended PDF version of this report, which includes individual report cards for each test combination that achieved a 'pass,' is available for free (to any Light Reading registered user) at this link.

    — Ray Le Maistre, Circle me on Google+ Follow me on TwitterVisit my LinkedIn profile, Editor-in-Chief, Light Reading and Carsten Rossenhövel, Managing Director, European Advanced Networking Test Center AG (EANTC) (http://www.eantc.de/), an independent test lab in Berlin. EANTC offers vendor-neutral network test facilities for manufacturers, service providers, and enterprises.

    Next page: The Participants

    (2)  | 
    Comment  | 
    Print  | 
  • Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
    User Rank: Light Beer
    12/10/2015 | 2:39:46 PM
    Network now needs a permanent Proof of Concept lab
    We're at the 'art of the possible' stage, rather than the 'start of the probable' regarding live deployment... which if taken at face value, should wholly sidestep the heinous blunder of productionising a nine 5's infrastructure in for a five 9's expectation.

    As much as the IT proposition to a user involves all layers of the ISO stack, if you screw the network you've lost the farm, which then negates any major investment in state of the art delivery systems up the stack.

    Any tech worth their weight will crash and burn this stuff on Proof of Concept lab... and weld all the fork doors closed... as it'll be their 'nads on the line should if fail in production, not any IT upper heirarchy who decided to make an (unwise) executive decision.

    Fail means *anything* regardless whether its a SNAFU or an hack-attack vector that no-one tested. 

    In short, test to destruction, armour-plate the resultant design, and always have a PoC lab bubbling away in the background for the next iteration, because the genie is now out of the bottle and won't go back in.
    User Rank: Light Sabre
    12/8/2015 | 8:56:52 PM
    Open source doesn't necessarily mean any support available...
    It's always interesting to see how open source platforms still have gaping holes with major unsupported functions. Every fork/flavor of an open source project means more than a few potential gaps for interoperability with other forks.

    Light Reading’s Upskill U is a FREE, interactive, online educational resource that delivers must-have education on themes that relate to the overall business transformation taking place in the communications industry.
    Wednesday, October 26, 1:00PM EDT
    Security: Tackling DDoS
    Gary Sockrider, Principal Security Technologist, Arbor Networks
    Friday, October 28, 1:00PM EDT
    Security: The Plusses and Minuses of Open Source Software
    Nick Feamster, Acting Director, Center for Information Technology Policy, Princeton University
    Wednesday, November 16, 1:00PM EST
    SDN 101
    John Isch, Practice Director, Network & Voice, Orange Business Services
    Friday, November 18, 1:00PM EST
    SDN & Open Source
    Christopher W. Rice, Senior Vice President of AT&T Labs, Domain 2.0 Architecture and Design
    in association with:
    From The Founder
    Light Reading today starts a new voyage as part of a larger Enterprise.
    Flash Poll
    Live Streaming Video
    Charting the CSP's Future
    Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
    LRTV Custom TV
    Next-Generation Technology Beyond DOCSIS 3.1

    10|20|16   |     |   (0) comments

    At SCTE 2016, Huawei's Liu Jianhua speaks with Alan Breznick for an exclusive interview.
    LRTV Custom TV
    Hybrid Video Solutions to Change TV, Change Future

    10|20|16   |     |   (0) comments

    At SCTE 2016, Huawei's Ian Locke speaks with Alan Breznick for an exclusive interview.
    LRTV Custom TV
    Huawei Future-Oriented Giga Coax Network

    10|20|16   |     |   (0) comments

    At SCTE 2016, Huawei's Allen Wang speaks with Alan Breznick for an exclusive interview.
    LRTV Custom TV
    Huawei at SCTE 2016

    10|20|16   |     |   (0) comments

    Join Alan Breznick of Light Reading and take a sneak peek at the Huawei booth at SCTE 2016.
    LRTV Custom TV
    Assuring Network Quality in a Rapidly Changing Environment

    10|20|16   |     |   (0) comments

    As the rate of change and complexity increases in agile networks, the importance of introducing DevOps methodologies for integrating active test and assurance solutions throughout the full service lifecycle becomes critical to ensure that customers are experiencing the service quality they demand. The industry landscape is changing, and software-based test and ...
    Telecom Innovators Video Showcase
    A10 Networks on Service Providers' Industry Needs

    10|20|16   |     |   (0) comments

    Light Reading's Steve Saunders hears how A10 enables service providers to accelerate, secure and optimize their application delivery to drive down costs, enhance service availability, and better respond to customer requirements, so they can improve customer satisfaction, monetize their network, and grow revenues.
    LRTV Custom TV
    New NFV Use Cases for Cable TV

    10|19|16   |     |   (0) comments

    A large number of NFV use cases are focused on the enterprise domain, looking at virtualization of customer-premises equipment (CPE). To date, there has been little focus on the use cases and business case for virtualization of the video content delivery networks required to deliver unicast and streaming video to consumers. Amdocs commissioned Analysys Mason to ...
    Women in Comms Introduction Videos
    Meet the Future Workforce: New Faces, Expectations & Motivations

    10|19|16   |   5:33   |   (1) comment

    Millennials and their younger peers, Gen Z, expect more out of their network and more out of their work. Intel's Lynn Comp shares how the industry can prepare for this new generation of workers.
    LRTV Custom TV
    ZTE Global Services User Congress 2016 Highlights

    10|19|16   |     |   (0) comments

    ZTE held its 2nd Global Service User Conference in Dusseldorf on October 13-14. Representatives from network operators, leading industry analysts and ZTE senior expertsattended the event, exploring the best practice in managed services and the vision to transform network operations into the operations center of the future (OpCF) in the software-defined networking ...
    LRTV Custom TV
    Cliff Grossner on Cloud & Network Synergy From Carrier Service

    10|18|16   |     |   (0) comments

    Local carriers offer the collaborated cloud and network service that benefits from their understanding of the regulations operating in different vertical markets.
In this interview, Cliff Grossner from IHS Technology talks about how this advanced service can support business agility and flexibility.
    LRTV Custom TV
    VeEX: Live from SCTE Cable-Tec Expo 2016

    10|17|16   |     |   (0) comments

    Cyrille Morelle, VeEX's President and CEO, talks with Light Reading's Alan Breznick live from the SCTE Cable-Tec Expo 2016. They discuss DOCSIS 3.1 technology, deployments and early lessons learned. New products on display include the CX350s-D3.1, CX380s-D3.1, CX310, AT2500-3G, FX150 OTDR and MTT WiFI Air Expert.
    LRTV Custom TV
    Smith Micro's Carrier-Grade WiFi Component

    10|17|16   |     |   (0) comments

    Join Carol Wilson of Light Reading as she interviews the CTO of Smith Micro, Dave Sperling.
    Upcoming Live Events
    November 3, 2016, The Montcalm Marble Arch, London
    November 30, 2016, The Westin Times Square, New York City
    December 1, 2016, The Westin Times Square, New York, NY
    December 6-8, 2016, The Westin Excelsior, Rome
    May 16-17, 2017, Austin Convention Center, Austin, TX
    All Upcoming Live Events
    Hot Topics
    Attacks Have Major Internet Sites on the Ropes
    Brian Santo, Senior editor, Test & Measurement / Components, Light Reading, 10/21/2016
    Verizon Wants More Data on Yahoo Breach
    Alan Breznick, Cable/Video Practice Leader, Light Reading, 10/20/2016
    Layer3 TV Comes to Town, Hints at Future
    Mari Silbey, Senior Editor, Cable/Video, 10/21/2016
    Oh Snap! Qualcomm Unveils X50, Its First 5G Modem
    Dan Jones, Mobile Editor, 10/17/2016
    WiCipedia: Grace Hopper Promotes Diversity, Girl Scouts Code & How to Thrive
    Eryn Leavens, Special Features & Copy Editor, 10/21/2016
    Like Us on Facebook
    Twitter Feed
    BETWEEN THE CEOs - Executive Interviews
    Join us for an in-depth interview between Steve Saunders of Light Reading and Alexis Black Bjorlin of Intel as they discuss the release of the company's Silicon Photonics platform, its performance, long-term prospects, customer expectations and much more.
    There's no question that, come 2020, 5G technology will turn the world's conception of what mobile networking is on its head. Within the world of 5G development, Dr. ...
    Animals with Phones
    Live Digital Audio

    A vital part of increasing the number of women in comms is transforming the ways companies can support and empower women. While progressive company policies that support both men and women in achieving work-life balance are a step in the right direction, creating a company culture that supports those policies can at times be more challenging.

    During this show, we'll talk to Lynn Comp, Senior Director of Industry and Sales Enabling (ISE) in the Network Platforms Group at Intel, about why those challenges exist and how companies can overcome them. She'll provide insight into how Intel has worked to create a culture that supports work-life balance, and provide steps and guidance for other companies wishing to do the same. We will also leave plenty of time to get your questions answered live on the air.