& cplSiteName &

2016 NIA-EANTC NFV Interoperability Test Report

Carsten Rossenhövel, Managing Director, EANTC
12/7/2016
50%
50%

Introduction
In the first year of its industry-wide interoperability program, the New IP Agency included 33 commercial solutions from 23 member companies. We conducted three test campaigns focusing on virtualized network functions (VNFs), the NFV infrastructure, VNF managers and NFV orchestrators. The tests identified more than eight general areas of interoperability issues seen today; they helped vendors to improve their solutions and resolve problems in multivendor scenarios not tested before. Vendor members have upstreamed test findings to open source projects such as OpenStack, and to standards-developing organizations such as ETSI.

NIA's place is to test multivendor interoperability industry-wide, independent of any specific commercial or non-commercial ecosystems, to ensure service providers can choose their implementations from the widest selection of offerings.

The number of NIA testing program participants has expanded over the year, and the success rates have improved gradually. This is great news -- but let's not get the wrong message: This industry is still in its early stages, and no success can be taken for granted unless tested individually.

Test coverage
The New IP Agency philosophy is to conduct established test campaigns pertinent to communication service providers' adoption and deployment of NFV solutions to broaden participation and improve interoperability in well-understood NFV areas such as basic life-cycle management.

Simultaneously, we aim to innovate with new test campaigns, accelerating interoperability of new developments and being among the first to validate as-yet unproven grounds. In both cases, we aim to conduct the most realistic, in-depth, and transparently documented public interoperability tests worldwide.

To this end, we have run our VNF-to-NFVi interoperability program throughout the course of 2016. Additionally, we complemented the VNF-to-NFVi evaluation with a one-time Service Chaining Orchestration interoperability showcase, hosted in June, and an initial VNF Manager-to-VIM interoperability campaign. Let's have a look at each of the three:

VNF-to-NFVi Interoperability Program
This first NIA testing program covers the basics: Image management and life-cycle management. We verify whether one party's VNF can be instantiated, configured, and terminated on an NFVi of another party. This may sound trivial; after some 50+ test combinations, we can assure the reader it is not. In a nutshell, interoperability issues show up because OpenStack implementations vary a lot, and most vendors have not (or not seriously) tested with each other before. (We cover the types of issues and their root causes further below.)

The tests include graceful and forceful VNF restart, scaling up (by increasing CPU capacity), and modifying network resources during operation.

VNF Manager-to-VIM Interoperability Campaign
The second NIA testing program began with a single campaign this year. It focuses on the interoperability between VNF managers and the virtual infrastructure management (VIM). The VNF manager's task is to manage all VNF compute, network and storage resources. VNF managers are specifically helpful to the deployment of complex VNFs consisting of multiple virtual machines (VMs) which are required to scale out and in, adding and removing resource instances such as VMs.

Our tests cover life-cycle management functions on the VNF level that the VNFM exposes to the northbound management and orchestration. In addition, we suggest resource migration, resource management, fault management and performance monitoring test cases.

For 2016, we decided not to include, or at least not focus on, the role of the NFV Orchestrator (NFVO) in this test program. The intent being to focus on VNFM/VIM interoperability tests as purely as possible. In addition, we aimed to focus on individual, VNF-specific VNFMs: These could potentially take scale decisions based on individual VNF (application-layer) conditions, and generally include element manager (EM) functions.

Service Chaining Orchestration Showcase
In June this year, NIA conducted a one-time, on-site interoperability test and showcase. The goal was to construct a complete NFV scenario including NFVi, VIM, VNFs and NFVO and showcase service chaining in a short time. We chose a representative virtualized CPE (vCPE) use case. The test plan included basic vCPE service creation, static and elastic bandwidth provisioning and service chaining with generic and flow-based forwarding graphs.

Participants
All NIA vendor members were eligible to participate, and 23 did -- equivalent to 90%. This year, a total of seven NFVis, three NFVOs and 23 VNFs participated.

Many vendors supported a subset of test campaigns, depending on their available resources, software readiness and -- in the case of NFVis -- hardware on-site.

All tests were carried out at EANTC's lab in Berlin, Germany with the exception of the orchestration campaign which was pre-staged and showcased in Austin, Texas.

Test Bed Topology
The test topology was fairly straightforward. NFVi vendors, with the support of testing solutions company Ixia and EANTC set up the hardware in EANTC's lab in Berlin. Each NFVi vendor provided its own hardware (for details see chart).

  • Cisco NFVi provided a full OpenStack deployment, including three redundant control nodes, three compute nodes, a redundant storage system, two build nodes and a router, plus a switch connecting the infrastructure internally until May 2016.

  • Starting in October, EANTC installed a Dell EMC NFVi. It is a full OpenStack environment, with three redundant control nodes, three compute nodes, three storage nodes, one platform manager, one control node and three switches (two leaf switches and a management switch). It also includes Dell EMC hardware element managers for deployment and management of the infrastructure.

  • Huawei provided FusionSphere with three hot-standby control nodes. The FusionSphere installation was upgraded in the summer and returned to EANTC with new hardware.

  • Juniper provided one Contrail server until April 2016 which implemented the OpenStack environment using the virtual deployment option: One virtual control node, one virtual compute node and a virtual router. This satisfied the minimum testing requirements in the first phase.

  • Nokia CloudBand provided a full OpenStack deployment for the whole year (three control and eight compute nodes including local storage, plus a router for internal connectivity).

  • ZTE provided Tulip Elastic Cloud System (TECS) from April onward. The TECS environment is a full OpenStack deployment using local storage to meet the NFVi testing requirements.
  • Overview of Results
    It is great news that most VNF/NFVi test combinations passed our tests in the December 2016 campaign. These include VNFs from Fortinet, Infoblox, Mitel and Procera running on NFVis from Dell EMC, Huawei, Nokia and ZTE -- 11 successful test combinations. Three combinations failed because of ex-ante incompatibilities between OpenStack versions. This makes a success rate of 79% for our latest campaign.

    Together with the earlier test campaign, 36 combinations passed in total and 17 failed or were not completed. The success rate across the year is 69%.

    Given the large number of potential test combinations and the limited testing time, EANTC did not aim for a full-mesh test campaign. The goal was to complete two test combinations per VNF, which worked out in some cases but not in others, often due to integration, configuration or troubleshooting delays. The solutions marked on the chart in blue were available until December 2016; the other vendors had not participated in the latest test campaign. Most cells left empty in the matrix above relate to test combinations that did not happen; very few relate to test combinations that failed. Fundamental interoperability issues are reported in the next section.

    Individual Findings and Issues
    Many of the findings described in this section are related to OpenStack -- kind of obvious as all participating vendors based their NFVi implementations on OpenStack. This is a great opportunity to align the industry’s nomenclature (all parties understand the term "heat template," for example) and to avoid reinventing the wheel (vendors equally benefit from OpenStack evolution).

    At the same time, it is a burden -- different implementations are based on OpenStack but are not identical. Executive management and marketing may overlook this fact -- but it is very important in practice. We had to adapt our automation framework for each single test combination. Vendors supplemented the OpenStack releases used in our test bed (Juno, Kilo and Liberty) more or less extensively, resulting in interoperability problems.

    These were mostly configuration problems that could be overcome during test preparation -- a big, positive difference to the first test campaign a year ago. As a result, test preparation took much longer than planned, in some cases literally weeks and weeks.

    At the OpenStack Summit in Barcelona in October, delegates mostly talked about Newton or Ocata release planning. It's unlikely service providers will adopt, or even have access to, these new releases with their advanced features any time soon. The nature of vendor development cycles and network deployment processes will always result in lagging behind OpenStack releases by around two years. It might be wise to react with a different release and support model for OpenStack just for telecom service providers.

    Here is a list of detailed findings areas:

    OpenStack Configuration
    EANTC created a single template for all NFVis to use. This helped us automate test execution and perform identical tests across multiple NFVis. The configurations were not always easy to implement -- we had to dig deep into the policy, tenants and topology configuration. Sometimes, issues are simple yet may result in time-consuming troubleshooting. Naming conventions were not always implemented correctly. Some NFVis did not like using underscore in their network names, for example.

    In the end, we had to handle each NFVi configuration separately. The conclusion remains similar to previous phases: NFVi configuration consumes time. Vendor implementations differ too much to be handled consistently.

    NFVi/VIM implementation bugs continued to bring us into trouble. We faced an issue where one of the vendor-supported NFVis lost database synchronization between its nodes which brought the whole NFVi to a halt. The NFVi had to be rebuilt from scratch. Luckily the NFVi vendor had a script to implement all configurations which meant little time was lost on recovering the NFVi. At the current maturity level, administrating and maintaining an OpenStack installation often requires directly interacting with the database.

    OpenStack Security
    We had no intentions to test security (yet) -- we just wanted everything to work smoothly in the closed environment of our lab. Vendor implementations of OpenStack security policies differ quite a bit, which often resulted in VNF data plane connectivity issues. VNFs that forwarded traffic on the Ethernet layer, i.e. were IP-transparent, were most affected.

    IP and MAC protection means that traffic cannot simply traverse a layer 2 virtual machine. This traffic will be dropped by default in some OpenStack implementations unless a security rule is updated on the virtual ports. EANTC tuned the test automation code to handle the different cases.

    OpenStack network policy improvements can potentially cause interruptions to existing VMs.

    Under the hood, our security issues were all related to different Neutron configurations. There is no easy way for a user to tell which features are installed and enabled, how they are configured, and how this will affect VNF functionality. Neutron is kind of a special case since both its use in general as well as the individual features it provides are entirely optional. While the "hardcore" OpenStack community is aware of Neutron usage, some vendors seemed to struggle with Neutron's philosophy.

    OpenStack Modules Version Consistency
    NFVi implementations based on OpenStack are quite modular. When creating a distribution, vendors can pick and choose from a range of components that the OpenStack community works on in different projects. Common sense would advise to choose a consistent set of software modules from a single OpenStack release. In a telecom environment, this approach fails:

  • Service providers want to run multi-vendor NFV solutions. It is unfeasible to wait with upgrades -- if only, for security reasons -- until all vendors involved in a virtualized network solution provide an upgrade to the same version.

  • Service providers want to keep their NFV infrastructure up and running all the time. They must implement in-service-rolling updates instead of updating all parts of a system at once.
  • As a result, telecom operators require multi-version interoperability. Unfortunately, this cannot be taken for granted. In the latest test round, we came across three major issues related to multi-version inconsistencies:

  • One of the participating NFVis used two versions of the OpenStack Identity API, as implemented by Keystone, in parallel: Version 2 for the NFVi's command line interface (CLI), and version 3 for the graphical user interface (GUI). Although EANTC's automation scripts -- obviously connecting to the CLI -- were fine using KeyStone v2, we could not use them easily: Users had been created via the GUI before, with an extended feature set of the newer KeyStone version, so the v2 CLI denied any subsequent activities. It turned out later that the vendor supported v3 over the CLI as well -- this problem was again a configuration/transparency issue.
  • In another case, we used the Horizon component of OpenStack to modify IP subnetwork allocation ranges. When applied to OpenStack Liberty versions of Horizon, this action worked as expected. When we applied the same command to one of the NFVis based on OpenStack Kilo, Horizon still reported success -- but did not execute the requested action. It seems nobody had thought about a proper negative response for "reserved for future use" unsupported requests.
  • Some VNFs under test implemented special Heat resources. They required the installation of those resources on the target NFVi. This added on the efforts to evaluate the VNF and introduced an additional version incompatibility with the NFVi. Although this incompatibility is rather simple to detect, it will be responsible for many interoperability issues and disappointments in the future. We recommend all VNF vendors should clearly communicate the minimum requirements for the software environment.
  • These are just three examples. Obviously, software behavior changes across versions; clients often need to be flexible when working with different versions of services. Unfortunately, we could not easily figure the exact versions of services/components in an NFVi installation. Distributors (vendors) would need to provide such information out of band as of today. We strongly recommend to implement better API version control/status operations in OpenStack-derived NFV components.

    Migrating VNFs: Not Really Trivial
    Migrating all or some VNF resources from one resource zone to another reliably is a non-trivial task which will continue to require a lot of focus. Specifically, network functions consisting of multiple components (virtual machines aka VNFCs) and the migration across multiple data center sites can be complex, as well as the resource calculation for the target zone.

    So far, we restricted our tests to migrating all compute resources of a VNF at once; most participating VNFs consisted of only a single VM anyway. We always migrated to a local target zone with sufficient resources.

    But still we got into problems with VNFs that process traffic on the Ethernet layer, i.e. are not routers. When resources of such VNFs were migrated, the Ethernet (MAC) address tables of connected switches did not get updated. Although the VNF resources had moved, traffic was still sent to the old compute node.

    Upgrading NFVis
    Participating vendors told us that an upgrade of NFVis using older OpenStack versions (Icehouse; Kilo) is a complex process only possible at the vendor location. This is one of the reasons why we have run all tests with software versions that are not officially maintained by OpenStack anymore, or maintenance will be discontinued in a few months. Multiple development stages are typically recognized for telecom software upgrades:

  • Manual upgrade by vendor (return equipment)

  • Field-serviceable manual upgrade by vendor with major downtime

  • Field-serviceable manual customer upgrade with major downtime

  • Automated customer serviceable upgrade with low downtime

  • In-service software upgrade with zero downtime.
  • Service providers want to get to the final stage quickly. The industry is currently at the first step for the OpenStack versions we have seen in our lab, and mentioned they plan to advance to the second step soon.

    VNF Manager Interoperability Tests

    We decided not to publish results of the VNFM/VIM interoperability test campaign. There were too many issues that might not be representative for the industry, and we would not have been able to reasonably anonymize our reporting.

    We will log the lessons learned and rerun the test campaign with newer software versions in the future.

    Summary
    A year ago, when we ran the first test campaign, many findings were new and it was difficult to overcome them. Vendors supplied new software in some cases; in 11 cases, we had to give up the test combination as not achievable. In the latest campaign, VNF and NFVi vendors generally understood the challenges better and provided more advanced software; most of the issues were related to configuration. Only three combinations failed because of major incompatibilities.

    Hotbed of Activity
    Test bed where EANTC put participating NIA member products through their interoperability paces.
    Test bed where EANTC put participating NIA member products through their interoperability paces.

    Still, the amount of prestaging time, the number of configuration questions, and the complexity of issues to overcome in each individual multi-vendor combination is very difficult to predict. Seemingly very experienced vendors can struggle with a tiny software engineering issue for days and weeks. In quite a few cases, the configuration issues were not even related to the multivendor aspect: They often had to do with basic misconfigurations of their own solution. Even though there were fewer "insurmountable" problems seen in the latest test campaign, the unpredictability is still a major issue.

    Our dream would be greatly streamlined open source solutions with reduced number of options, improved installation/configuration procedures and more extensive telecom focused baseline software in OpenStack. We hope that the industry might understand open source does not substitute standards at all, and will develop normative APIs very soon. In addition, much better software documentation and more readily available education programs for support engineers would be high on our wish list.

    Outlook
    The journey continues: It is noticeable that vendors have kept up the pace of development and have a lot of plans for management and orchestration support in 2017. The NIA Board has approved a test campaign covering NFV orchestration and VNF manager to VIM interoperability for the first quarter of 2017. It will be followed by a multi-vendor SDN and NFV integration test in the second quarter.

    The New IP Agency intentionally does not have a technical committee developing test plans. We plan to align the Q1 test session with ETSI's draft interop test specification TST007, being developed by EANTC and other ETSI members right now. Any test plan insights will be fed back to ETSI.

    In parallel, we will continue to further automate the existing test programs, most specifically the VNF to NFVi interoperability effort. The test plans will be amended by requests from service provider members. Our goal is to continually expand the tests with realistic cases, reducing service provider effort for baselining the interoperability status of solutions they intend to purchase.

    — Carsten Rossenhövel is Managing Director of the European Advanced Networking Test Center (EANTC), which conducted the tests for the New IP Agency. Special to The New IP


    CALLING ALL TEST, ASSURANCE AND MONITORING COMPANIES:
    Make sure your company and services are listed free of charge at Testapedia, the comprehensive set of searchable databases covering the companies, products, industry organizations and people that are directly involved in defining and shaping the telecom test and measurement industry.


    (0)  | 
    Comment  | 
    Print  | 
    Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
    From The Founder
    Kicking off BCE 2017, Light Reading founder Steve Saunders lays blame for NFV's slow ramp-up and urges telecom to return to old-fashioned standards building and interoperability.
    Flash Poll
    Live Streaming Video
    Charting the CSP's Future
    Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
    LRTV Interviews
    CenturyLink: Let's Get Past SD-WAN Hype

    6|23|17   |   04:02   |   (0) comments


    Technology becomes a "shiny object" unless it's properly focused on solving business needs for enterprise customers, says Bill Grubbs, network solutions architect for CenturyLink. He explains to Light Reading why SD-WAN deployments have to be tailored to specific needs – and more.
    Women in Comms Introduction Videos
    Infinera's Sales Director Paints Tech's Big Picture

    6|21|17   |   4:14   |   (1) comment


    Shannon Williams, Infinera's director of sales, shares how she achieves work's many balancing acts -- between her role and the broader company, today and tomorrow's tech and more.
    LRTV Custom TV
    SD-WAN Innovation & Trends

    6|20|17   |     |   (0) comments


    Versa CEO Kelly Ahuja discusses with Carol Wilson the current status and trends in the SD-WAN market, Versa's innovation around building a software platform with broad contextualization, and the advantages that startups can bring to the SD-WAN market.
    LRTV Interviews
    Ovum's Dario Talmesio on 5G in Europe

    6|20|17   |   02:16   |   (0) comments


    At 5G World 2017, Dario Talmesio, principal analyst and practice leader on Ovum's fixed and mobile telecoms European team, explains the emerging trends amongst European operators as they prepare for 5G.
    LRTV Custom TV
    Putting Power on a Pedestal

    6|19|17   |     |   (0) comments


    ARRIS's John Ulm says a major accomplishment of SCTE•ISBE's Energy 2020 program is increased focus on power cost and consumption, including inclusion of energy requirements in operators' RFPs and RFIs.
    LRTV Custom TV
    Gigabit Access: The Last-Mile Pipe for All Future Services

    6|19|17   |     |   (0) comments


    A Gigabit access platform being deployed today must be able to deliver all types of services to an increasing number of devices. A non-blocking architecture is necessary to support the ever-increasing growth in bandwidth demand. The Huawei Gigabit access solution is based on a distributed design that is fully scalable to deliver a unprecedented performance.
    LRTV Custom TV
    Key Factors to Successfully Deploy an SD-WAN Service

    6|19|17   |     |   (0) comments


    As service providers transition their SD-WAN solution from trials and limited deployments into production at large scale, there are important considerations to successfully operationalize these solutions and realize their full potential, without adding complexity, introducing uncertainty or disrupting current business operations. Sunil Khandekar, CEO and Founder ...
    LRTV Custom TV
    IoT Solutions: Rational Exuberance

    6|19|17   |     |   (0) comments


    IoT solutions are morphing from hype into viable business opportunities. Huawei has the platform and ecosystem support to help carriers successfully address new business opportunities in the IoT space.
    LRTV Custom TV
    Realizing ICN as a Network Slice for Mobile Data Distribution

    6|19|17   |     |   (1) comment


    Network slicing in 5G allows the potential introduction of new network architectures such as Information-centric Networks (ICN) as a slice, managed over a shared pool of compute, storage and bandwidth resource. Services over an ICN slice can benefit from many architectural features such as Name Based Networking, Security, Multicasting, Multi-homing, Mobility, ...
    LRTV Interviews
    Ovum's Mike Roberts on 5G Uptake

    6|19|17   |   04:08   |   (0) comments


    Mike Roberts, research director for Ovum's service provider markets group, explains why he has boosted his 5G subscriptions forecast.
    LRTV Interviews
    AT&T's Hubbard on Intersection of SD-WAN & MPLS

    6|15|17   |     |   (0) comments


    Rick Hubbard, SVP of Network Product Management for AT&T Business Solutions, discusses how AT&T's approach to SD-WAN fits in with its overall virtualization strategy, explains how SD-WAN can improve enterprise customers' use of the cloud and addresses the intersection of SD-WAN and MPLS.
    Telecom Innovators Video Showcase
    Keep Connected IoT Devices Under Control With Allot

    6|15|17   |     |   (0) comments


    Allot AVP of International Pre-Sales, Daniel Keidar, explains how communications service providers can protect infrastructure and service availability from flooding attacks caused by malfunctioning or bot-infected devices connected to their network.
    Upcoming Live Events
    October 18, 2017, Colorado Convention Center - Denver, CO
    November 1, 2017, The Montcalm Marble Arch
    November 1, 2017, The Montcalm Marble Arch
    November 30, 2017, The Westin Times Square
    All Upcoming Live Events
    Infographics
    With the mobile ecosystem becoming increasingly vulnerable to security threats, AdaptiveMobile has laid out some of the key considerations for the wireless community.
    Hot Topics
    Netflix's Lesson in Culture Expectation Settings
    Sarah Thomas, Director, Women in Comms, 6/21/2017
    No Imagination: UK Chip Biz Goes Up for Sale
    Iain Morris, News Editor, 6/22/2017
    Kalanick Steps Down as Uber CEO
    Sarah Thomas, Director, Women in Comms, 6/21/2017
    BT Tech Chief Makes Plea to 5G Chip Vendors
    Ray Le Maistre, International Group Editor, 6/20/2017
    Like Us on Facebook
    Twitter Feed
    BETWEEN THE CEOs - Executive Interviews
    Following a recent board meeting, the New IP Agency (NIA) has a new strategy to help accelerate the adoption of NFV capabilities, explains the Agency's Founder and Secretary, Steve Saunders.
    One of the nice bits of my job (other than the teeny tiny salary, obviously) is that I get to pick and choose who I interview for this slot on the Light Reading home ...
    Animals with Phones
    Live Digital Audio

    Playing it safe can only get you so far. Sometimes the biggest bets have the biggest payouts, and that is true in your career as well. For this radio show, Caroline Chan, general manager of the 5G Infrastructure Division of the Network Platform Group at Intel, will share her own personal story of how she successfully took big bets to build a successful career, as well as offer advice on how you can do the same. We’ll cover everything from how to overcome fear and manage risk, how to be prepared for where technology is going in the future and how to structure your career in a way to ensure you keep progressing. Chan, a seasoned telecom veteran and effective risk taker herself, will also leave plenty of time to answer all your questions live on the air.