& cplSiteName &

2016 NIA-EANTC NFV Interoperability Test Report

Carsten Rossenhövel, Managing Director, EANTC
12/7/2016
50%
50%

Introduction
In the first year of its industry-wide interoperability program, the New IP Agency included 33 commercial solutions from 23 member companies. We conducted three test campaigns focusing on virtualized network functions (VNFs), the NFV infrastructure, VNF managers and NFV orchestrators. The tests identified more than eight general areas of interoperability issues seen today; they helped vendors to improve their solutions and resolve problems in multivendor scenarios not tested before. Vendor members have upstreamed test findings to open source projects such as OpenStack, and to standards-developing organizations such as ETSI.

NIA's place is to test multivendor interoperability industry-wide, independent of any specific commercial or non-commercial ecosystems, to ensure service providers can choose their implementations from the widest selection of offerings.

The number of NIA testing program participants has expanded over the year, and the success rates have improved gradually. This is great news -- but let's not get the wrong message: This industry is still in its early stages, and no success can be taken for granted unless tested individually.

Test coverage
The New IP Agency philosophy is to conduct established test campaigns pertinent to communication service providers' adoption and deployment of NFV solutions to broaden participation and improve interoperability in well-understood NFV areas such as basic life-cycle management.

Simultaneously, we aim to innovate with new test campaigns, accelerating interoperability of new developments and being among the first to validate as-yet unproven grounds. In both cases, we aim to conduct the most realistic, in-depth, and transparently documented public interoperability tests worldwide.

To this end, we have run our VNF-to-NFVi interoperability program throughout the course of 2016. Additionally, we complemented the VNF-to-NFVi evaluation with a one-time Service Chaining Orchestration interoperability showcase, hosted in June, and an initial VNF Manager-to-VIM interoperability campaign. Let's have a look at each of the three:

VNF-to-NFVi Interoperability Program
This first NIA testing program covers the basics: Image management and life-cycle management. We verify whether one party's VNF can be instantiated, configured, and terminated on an NFVi of another party. This may sound trivial; after some 50+ test combinations, we can assure the reader it is not. In a nutshell, interoperability issues show up because OpenStack implementations vary a lot, and most vendors have not (or not seriously) tested with each other before. (We cover the types of issues and their root causes further below.)

The tests include graceful and forceful VNF restart, scaling up (by increasing CPU capacity), and modifying network resources during operation.

VNF Manager-to-VIM Interoperability Campaign
The second NIA testing program began with a single campaign this year. It focuses on the interoperability between VNF managers and the virtual infrastructure management (VIM). The VNF manager's task is to manage all VNF compute, network and storage resources. VNF managers are specifically helpful to the deployment of complex VNFs consisting of multiple virtual machines (VMs) which are required to scale out and in, adding and removing resource instances such as VMs.

Our tests cover life-cycle management functions on the VNF level that the VNFM exposes to the northbound management and orchestration. In addition, we suggest resource migration, resource management, fault management and performance monitoring test cases.

For 2016, we decided not to include, or at least not focus on, the role of the NFV Orchestrator (NFVO) in this test program. The intent being to focus on VNFM/VIM interoperability tests as purely as possible. In addition, we aimed to focus on individual, VNF-specific VNFMs: These could potentially take scale decisions based on individual VNF (application-layer) conditions, and generally include element manager (EM) functions.

Service Chaining Orchestration Showcase
In June this year, NIA conducted a one-time, on-site interoperability test and showcase. The goal was to construct a complete NFV scenario including NFVi, VIM, VNFs and NFVO and showcase service chaining in a short time. We chose a representative virtualized CPE (vCPE) use case. The test plan included basic vCPE service creation, static and elastic bandwidth provisioning and service chaining with generic and flow-based forwarding graphs.

Participants
All NIA vendor members were eligible to participate, and 23 did -- equivalent to 90%. This year, a total of seven NFVis, three NFVOs and 23 VNFs participated.

Many vendors supported a subset of test campaigns, depending on their available resources, software readiness and -- in the case of NFVis -- hardware on-site.

All tests were carried out at EANTC's lab in Berlin, Germany with the exception of the orchestration campaign which was pre-staged and showcased in Austin, Texas.

Test Bed Topology
The test topology was fairly straightforward. NFVi vendors, with the support of testing solutions company Ixia and EANTC set up the hardware in EANTC's lab in Berlin. Each NFVi vendor provided its own hardware (for details see chart).

  • Cisco NFVi provided a full OpenStack deployment, including three redundant control nodes, three compute nodes, a redundant storage system, two build nodes and a router, plus a switch connecting the infrastructure internally until May 2016.

  • Starting in October, EANTC installed a Dell EMC NFVi. It is a full OpenStack environment, with three redundant control nodes, three compute nodes, three storage nodes, one platform manager, one control node and three switches (two leaf switches and a management switch). It also includes Dell EMC hardware element managers for deployment and management of the infrastructure.

  • Huawei provided FusionSphere with three hot-standby control nodes. The FusionSphere installation was upgraded in the summer and returned to EANTC with new hardware.

  • Juniper provided one Contrail server until April 2016 which implemented the OpenStack environment using the virtual deployment option: One virtual control node, one virtual compute node and a virtual router. This satisfied the minimum testing requirements in the first phase.

  • Nokia CloudBand provided a full OpenStack deployment for the whole year (three control and eight compute nodes including local storage, plus a router for internal connectivity).

  • ZTE provided Tulip Elastic Cloud System (TECS) from April onward. The TECS environment is a full OpenStack deployment using local storage to meet the NFVi testing requirements.
  • Overview of Results
    It is great news that most VNF/NFVi test combinations passed our tests in the December 2016 campaign. These include VNFs from Fortinet, Infoblox, Mitel and Procera running on NFVis from Dell EMC, Huawei, Nokia and ZTE -- 11 successful test combinations. Three combinations failed because of ex-ante incompatibilities between OpenStack versions. This makes a success rate of 79% for our latest campaign.

    Together with the earlier test campaign, 36 combinations passed in total and 17 failed or were not completed. The success rate across the year is 69%.

    Given the large number of potential test combinations and the limited testing time, EANTC did not aim for a full-mesh test campaign. The goal was to complete two test combinations per VNF, which worked out in some cases but not in others, often due to integration, configuration or troubleshooting delays. The solutions marked on the chart in blue were available until December 2016; the other vendors had not participated in the latest test campaign. Most cells left empty in the matrix above relate to test combinations that did not happen; very few relate to test combinations that failed. Fundamental interoperability issues are reported in the next section.

    Individual Findings and Issues
    Many of the findings described in this section are related to OpenStack -- kind of obvious as all participating vendors based their NFVi implementations on OpenStack. This is a great opportunity to align the industry’s nomenclature (all parties understand the term "heat template," for example) and to avoid reinventing the wheel (vendors equally benefit from OpenStack evolution).

    At the same time, it is a burden -- different implementations are based on OpenStack but are not identical. Executive management and marketing may overlook this fact -- but it is very important in practice. We had to adapt our automation framework for each single test combination. Vendors supplemented the OpenStack releases used in our test bed (Juno, Kilo and Liberty) more or less extensively, resulting in interoperability problems.

    These were mostly configuration problems that could be overcome during test preparation -- a big, positive difference to the first test campaign a year ago. As a result, test preparation took much longer than planned, in some cases literally weeks and weeks.

    At the OpenStack Summit in Barcelona in October, delegates mostly talked about Newton or Ocata release planning. It's unlikely service providers will adopt, or even have access to, these new releases with their advanced features any time soon. The nature of vendor development cycles and network deployment processes will always result in lagging behind OpenStack releases by around two years. It might be wise to react with a different release and support model for OpenStack just for telecom service providers.

    Here is a list of detailed findings areas:

    OpenStack Configuration
    EANTC created a single template for all NFVis to use. This helped us automate test execution and perform identical tests across multiple NFVis. The configurations were not always easy to implement -- we had to dig deep into the policy, tenants and topology configuration. Sometimes, issues are simple yet may result in time-consuming troubleshooting. Naming conventions were not always implemented correctly. Some NFVis did not like using underscore in their network names, for example.

    In the end, we had to handle each NFVi configuration separately. The conclusion remains similar to previous phases: NFVi configuration consumes time. Vendor implementations differ too much to be handled consistently.

    NFVi/VIM implementation bugs continued to bring us into trouble. We faced an issue where one of the vendor-supported NFVis lost database synchronization between its nodes which brought the whole NFVi to a halt. The NFVi had to be rebuilt from scratch. Luckily the NFVi vendor had a script to implement all configurations which meant little time was lost on recovering the NFVi. At the current maturity level, administrating and maintaining an OpenStack installation often requires directly interacting with the database.

    OpenStack Security
    We had no intentions to test security (yet) -- we just wanted everything to work smoothly in the closed environment of our lab. Vendor implementations of OpenStack security policies differ quite a bit, which often resulted in VNF data plane connectivity issues. VNFs that forwarded traffic on the Ethernet layer, i.e. were IP-transparent, were most affected.

    IP and MAC protection means that traffic cannot simply traverse a layer 2 virtual machine. This traffic will be dropped by default in some OpenStack implementations unless a security rule is updated on the virtual ports. EANTC tuned the test automation code to handle the different cases.

    OpenStack network policy improvements can potentially cause interruptions to existing VMs.

    Under the hood, our security issues were all related to different Neutron configurations. There is no easy way for a user to tell which features are installed and enabled, how they are configured, and how this will affect VNF functionality. Neutron is kind of a special case since both its use in general as well as the individual features it provides are entirely optional. While the "hardcore" OpenStack community is aware of Neutron usage, some vendors seemed to struggle with Neutron's philosophy.

    OpenStack Modules Version Consistency
    NFVi implementations based on OpenStack are quite modular. When creating a distribution, vendors can pick and choose from a range of components that the OpenStack community works on in different projects. Common sense would advise to choose a consistent set of software modules from a single OpenStack release. In a telecom environment, this approach fails:

  • Service providers want to run multi-vendor NFV solutions. It is unfeasible to wait with upgrades -- if only, for security reasons -- until all vendors involved in a virtualized network solution provide an upgrade to the same version.

  • Service providers want to keep their NFV infrastructure up and running all the time. They must implement in-service-rolling updates instead of updating all parts of a system at once.
  • As a result, telecom operators require multi-version interoperability. Unfortunately, this cannot be taken for granted. In the latest test round, we came across three major issues related to multi-version inconsistencies:

  • One of the participating NFVis used two versions of the OpenStack Identity API, as implemented by Keystone, in parallel: Version 2 for the NFVi's command line interface (CLI), and version 3 for the graphical user interface (GUI). Although EANTC's automation scripts -- obviously connecting to the CLI -- were fine using KeyStone v2, we could not use them easily: Users had been created via the GUI before, with an extended feature set of the newer KeyStone version, so the v2 CLI denied any subsequent activities. It turned out later that the vendor supported v3 over the CLI as well -- this problem was again a configuration/transparency issue.
  • In another case, we used the Horizon component of OpenStack to modify IP subnetwork allocation ranges. When applied to OpenStack Liberty versions of Horizon, this action worked as expected. When we applied the same command to one of the NFVis based on OpenStack Kilo, Horizon still reported success -- but did not execute the requested action. It seems nobody had thought about a proper negative response for "reserved for future use" unsupported requests.
  • Some VNFs under test implemented special Heat resources. They required the installation of those resources on the target NFVi. This added on the efforts to evaluate the VNF and introduced an additional version incompatibility with the NFVi. Although this incompatibility is rather simple to detect, it will be responsible for many interoperability issues and disappointments in the future. We recommend all VNF vendors should clearly communicate the minimum requirements for the software environment.
  • These are just three examples. Obviously, software behavior changes across versions; clients often need to be flexible when working with different versions of services. Unfortunately, we could not easily figure the exact versions of services/components in an NFVi installation. Distributors (vendors) would need to provide such information out of band as of today. We strongly recommend to implement better API version control/status operations in OpenStack-derived NFV components.

    Migrating VNFs: Not Really Trivial
    Migrating all or some VNF resources from one resource zone to another reliably is a non-trivial task which will continue to require a lot of focus. Specifically, network functions consisting of multiple components (virtual machines aka VNFCs) and the migration across multiple data center sites can be complex, as well as the resource calculation for the target zone.

    So far, we restricted our tests to migrating all compute resources of a VNF at once; most participating VNFs consisted of only a single VM anyway. We always migrated to a local target zone with sufficient resources.

    But still we got into problems with VNFs that process traffic on the Ethernet layer, i.e. are not routers. When resources of such VNFs were migrated, the Ethernet (MAC) address tables of connected switches did not get updated. Although the VNF resources had moved, traffic was still sent to the old compute node.

    Upgrading NFVis
    Participating vendors told us that an upgrade of NFVis using older OpenStack versions (Icehouse; Kilo) is a complex process only possible at the vendor location. This is one of the reasons why we have run all tests with software versions that are not officially maintained by OpenStack anymore, or maintenance will be discontinued in a few months. Multiple development stages are typically recognized for telecom software upgrades:

  • Manual upgrade by vendor (return equipment)

  • Field-serviceable manual upgrade by vendor with major downtime

  • Field-serviceable manual customer upgrade with major downtime

  • Automated customer serviceable upgrade with low downtime

  • In-service software upgrade with zero downtime.
  • Service providers want to get to the final stage quickly. The industry is currently at the first step for the OpenStack versions we have seen in our lab, and mentioned they plan to advance to the second step soon.

    VNF Manager Interoperability Tests

    We decided not to publish results of the VNFM/VIM interoperability test campaign. There were too many issues that might not be representative for the industry, and we would not have been able to reasonably anonymize our reporting.

    We will log the lessons learned and rerun the test campaign with newer software versions in the future.

    Summary
    A year ago, when we ran the first test campaign, many findings were new and it was difficult to overcome them. Vendors supplied new software in some cases; in 11 cases, we had to give up the test combination as not achievable. In the latest campaign, VNF and NFVi vendors generally understood the challenges better and provided more advanced software; most of the issues were related to configuration. Only three combinations failed because of major incompatibilities.

    Hotbed of Activity
    Test bed where EANTC put participating NIA member products through their interoperability paces.
    Test bed where EANTC put participating NIA member products through their interoperability paces.

    Still, the amount of prestaging time, the number of configuration questions, and the complexity of issues to overcome in each individual multi-vendor combination is very difficult to predict. Seemingly very experienced vendors can struggle with a tiny software engineering issue for days and weeks. In quite a few cases, the configuration issues were not even related to the multivendor aspect: They often had to do with basic misconfigurations of their own solution. Even though there were fewer "insurmountable" problems seen in the latest test campaign, the unpredictability is still a major issue.

    Our dream would be greatly streamlined open source solutions with reduced number of options, improved installation/configuration procedures and more extensive telecom focused baseline software in OpenStack. We hope that the industry might understand open source does not substitute standards at all, and will develop normative APIs very soon. In addition, much better software documentation and more readily available education programs for support engineers would be high on our wish list.

    Outlook
    The journey continues: It is noticeable that vendors have kept up the pace of development and have a lot of plans for management and orchestration support in 2017. The NIA Board has approved a test campaign covering NFV orchestration and VNF manager to VIM interoperability for the first quarter of 2017. It will be followed by a multi-vendor SDN and NFV integration test in the second quarter.

    The New IP Agency intentionally does not have a technical committee developing test plans. We plan to align the Q1 test session with ETSI's draft interop test specification TST007, being developed by EANTC and other ETSI members right now. Any test plan insights will be fed back to ETSI.

    In parallel, we will continue to further automate the existing test programs, most specifically the VNF to NFVi interoperability effort. The test plans will be amended by requests from service provider members. Our goal is to continually expand the tests with realistic cases, reducing service provider effort for baselining the interoperability status of solutions they intend to purchase.

    — Carsten Rossenhövel is Managing Director of the European Advanced Networking Test Center (EANTC), which conducted the tests for the New IP Agency. Special to The New IP


    CALLING ALL TEST, ASSURANCE AND MONITORING COMPANIES:
    Make sure your company and services are listed free of charge at Testapedia, the comprehensive set of searchable databases covering the companies, products, industry organizations and people that are directly involved in defining and shaping the telecom test and measurement industry.


    (0)  | 
    Comment  | 
    Print  | 
    Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
    From The Founder
    NFV's promises of automation and virtualization are intriguing, but what really excites service providers is the massive amount of money they could save.
    Flash Poll
    Live Streaming Video
    Charting the CSP's Future
    Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
    Women in Comms Introduction Videos
    VMWare VP Brings Women Up With Her

    8|16|17   |   6:49   |   (1) comment


    It's an art and a science to make mentorship, inclusive leadership, diversity and promotion of high-potential women work, says Honore' LaBourdette, vice president of Global Market Development at VMWare.
    LRTV Documentaries
    5G Spectrum Wars – The Recap

    8|15|17   |   2:22   |   (0) comments


    Service provider 3 has filed a lawsuit against Ofcom over 5G spectrum auction in the UK.
    LRTV Custom TV
    Say What? Facebook Unleashes AI Anarchy – The Recap

    8|7|17   |     |   (0) comments


    A recap of the week's talking points on Light Reading's sister site, telecoms.com. Facebook AI programmers had a bit of a brain-fade as they allowed one of its AI applications to invent its ...
    Women in Comms Introduction Videos
    Fujitsu's Women Band Together to Help Girls Do STEM

    8|2|17   |   9:35   |   (1) comment


    Supporting women both inside and outside of Fujitsu is a top priority of the telecom vendor. Yanbing Li, Fujitsu Network Communication's director of System Software Development & Delivery, shares why it's important, but why there's still a long road ahead.
    LRTV Custom TV
    If You're Not First, You're Last – The Recap

    7|31|17   |   08:18   |   (1) comment


    In case you missed it, Amazon's 1% stock increase helped Jeff Bezos dethrone Bill Gates as the richest man in the world. Also, Taiwanese electronics manufacturer
    Women in Comms Introduction Videos
    AT&T's Tech President Preps Workforce for the Future

    7|26|17   |   5:47   |   (10) comments


    AT&T is focused on the software-defined network of the future and is reskilling its workforce to get ready too, according to AT&T's President of Technology Development Melissa Arnoldi.
    Women in Comms Introduction Videos
    Cisco: Mentoring Critical to Attract & Retain Women

    7|19|17   |   6:40   |   (1) comment


    Liz Centoni, senior vice president and general manager of Cisco's Computing System Product Group, shares why mentoring in all its forms is important for women and what Cisco is doing that's made a difference for women in tech.
    LRTV Custom TV
    Gigabit LTE With Snapdragon 835

    7|12|17   |     |   (1) comment


    At an event in Wembley stadium, EE used its live network to demonstrate gigabit LTE using a Sony Xperia XZ Premium smartphone with a Qualcomm Snapdragon 835 chip.
    LRTV Custom TV
    Implementing Machine Intelligence With Guavus

    7|12|17   |     |   (0) comments


    Guavus unites big data and machine intelligence, enabling many of the the largest service providers in the world to save money and drive measureable revenue. Learn how applying Machine Intelligence substantially reduces operational costs and in many cases can eliminate subscriber impact, meaning a better subscriber experience and higher NPS.
    LRTV Custom TV
    Unlocking Customer Experience Insights With Machine Intelligence

    7|12|17   |     |   (0) comments


    When used to analyze operational data and to drive operational decisions, machine intelligence reduces the number of tasks which require human intervention. Guavus invested in Machine Intelligence early. Learn about the difference between Machine Learning and Machine Intelligence.
    Women in Comms Introduction Videos
    Verizon VP Talks Network, Career Planning

    7|12|17   |   4:49   |   (0) comments


    Heidi Hemmer, vice president of Technology, Strategy & Planning at Verizon, shares how bold bets and the future of tech define her career.
    Telecom Innovators Video Showcase
    Masergy's NFV Journey

    7|11|17   |     |   (0) comments


    Ray Watson, vice president of global technology at Masergy, discusses the advantages and challenges in entering the still-maturing NFV market for the past three years.
    Upcoming Live Events
    September 28, 2017, Denver, CO
    October 18, 2017, Colorado Convention Center - Denver, CO
    November 1, 2017, The Royal Garden Hotel
    November 1, 2017, The Montcalm Marble Arch
    November 2, 2017, 8 Northumberland Avenue, London, UK
    November 30, 2017, The Westin Times Square
    All Upcoming Live Events
    Infographics
    With the mobile ecosystem becoming increasingly vulnerable to security threats, AdaptiveMobile has laid out some of the key considerations for the wireless community.
    Hot Topics
    Intel CEO Leaves Trump Biz Advisory Board
    Dan Jones, Mobile Editor, 8/15/2017
    Are Cord-Cutting's Days Numbered?
    Alan Breznick, Cable/Video Practice Leader, Light Reading, 8/14/2017
    Orchestration Startup UBiqube Pivots Away From NFV
    Carol Wilson, Editor-at-large, 8/15/2017
    Verizon Video Woes Pile On
    Mari Silbey, Senior Editor, Cable/Video, 8/14/2017
    WiCipedia: Dolly Babes, Manifesto Backlash & 'Brotastic' Failures
    Eryn Leavens, Special Features & Copy Editor, 8/18/2017
    Like Us on Facebook
    Twitter Feed
    Animals with Phones
    We Know a Tough Day When We See One Click Here
    Live Digital Audio

    Understanding the full experience of women in technology requires starting at the collegiate level (or sooner) and studying the technologies women are involved with, company cultures they're part of and personal experiences of individuals.

    During this WiC radio show, we will talk with Nicole Engelbert, the director of Research & Analysis for Ovum Technology and a 23-year telecom industry veteran, about her experiences and perspectives on women in tech. Engelbert covers infrastructure, applications and industries for Ovum, but she is also involved in the research firm's higher education team and has helped colleges and universities globally leverage technology as a strategy for improving recruitment, retention and graduation performance.

    She will share her unique insight into the collegiate level, where women pursuing engineering and STEM-related degrees is dwindling. Engelbert will also reveal new, original Ovum research on the topics of artificial intelligence, the Internet of Things, security and augmented reality, as well as discuss what each of those technologies might mean for women in our field. As always, we'll also leave plenty of time to answer all your questions live on the air and chat board.