New report recaps NIA's evaluations covering VNFs, NFVis and Service Chaining and several new vendor combinations with Dell EMC, Fortinet, Huawei, Infoblox, Mitel, Nokia, Procera, Nokia and ZTE that were hosted as part of the NIA's VNF and NFVi evaluation.

December 7, 2016

16 Min Read
2016 NIA-EANTC NFV Interoperability Test Report

Introduction
In the first year of its industry-wide interoperability program, the New IP Agency included 33 commercial solutions from 23 member companies. We conducted three test campaigns focusing on virtualized network functions (VNFs), the NFV infrastructure, VNF managers and NFV orchestrators. The tests identified more than eight general areas of interoperability issues seen today; they helped vendors to improve their solutions and resolve problems in multivendor scenarios not tested before. Vendor members have upstreamed test findings to open source projects such as OpenStack, and to standards-developing organizations such as ETSI.

NIA's place is to test multivendor interoperability industry-wide, independent of any specific commercial or non-commercial ecosystems, to ensure service providers can choose their implementations from the widest selection of offerings.

The number of NIA testing program participants has expanded over the year, and the success rates have improved gradually. This is great news -- but let's not get the wrong message: This industry is still in its early stages, and no success can be taken for granted unless tested individually.

Test coverage
The New IP Agency philosophy is to conduct established test campaigns pertinent to communication service providers' adoption and deployment of NFV solutions to broaden participation and improve interoperability in well-understood NFV areas such as basic life-cycle management.

Simultaneously, we aim to innovate with new test campaigns, accelerating interoperability of new developments and being among the first to validate as-yet unproven grounds. In both cases, we aim to conduct the most realistic, in-depth, and transparently documented public interoperability tests worldwide.

To this end, we have run our VNF-to-NFVi interoperability program throughout the course of 2016. Additionally, we complemented the VNF-to-NFVi evaluation with a one-time Service Chaining Orchestration interoperability showcase, hosted in June, and an initial VNF Manager-to-VIM interoperability campaign. Let's have a look at each of the three:

VNF-to-NFVi Interoperability Program
This first NIA testing program covers the basics: Image management and life-cycle management. We verify whether one party's VNF can be instantiated, configured, and terminated on an NFVi of another party. This may sound trivial; after some 50+ test combinations, we can assure the reader it is not. In a nutshell, interoperability issues show up because OpenStack implementations vary a lot, and most vendors have not (or not seriously) tested with each other before. (We cover the types of issues and their root causes further below.)

The tests include graceful and forceful VNF restart, scaling up (by increasing CPU capacity), and modifying network resources during operation.

VNF Manager-to-VIM Interoperability Campaign
The second NIA testing program began with a single campaign this year. It focuses on the interoperability between VNF managers and the virtual infrastructure management (VIM). The VNF manager's task is to manage all VNF compute, network and storage resources. VNF managers are specifically helpful to the deployment of complex VNFs consisting of multiple virtual machines (VMs) which are required to scale out and in, adding and removing resource instances such as VMs.

Our tests cover life-cycle management functions on the VNF level that the VNFM exposes to the northbound management and orchestration. In addition, we suggest resource migration, resource management, fault management and performance monitoring test cases.

For 2016, we decided not to include, or at least not focus on, the role of the NFV Orchestrator (NFVO) in this test program. The intent being to focus on VNFM/VIM interoperability tests as purely as possible. In addition, we aimed to focus on individual, VNF-specific VNFMs: These could potentially take scale decisions based on individual VNF (application-layer) conditions, and generally include element manager (EM) functions.

Service Chaining Orchestration Showcase
In June this year, NIA conducted a one-time, on-site interoperability test and showcase. The goal was to construct a complete NFV scenario including NFVi, VIM, VNFs and NFVO and showcase service chaining in a short time. We chose a representative virtualized CPE (vCPE) use case. The test plan included basic vCPE service creation, static and elastic bandwidth provisioning and service chaining with generic and flow-based forwarding graphs.

Participants
All NIA vendor members were eligible to participate, and 23 did -- equivalent to 90%. This year, a total of seven NFVis, three NFVOs and 23 VNFs participated.

Figure 1:

Many vendors supported a subset of test campaigns, depending on their available resources, software readiness and -- in the case of NFVis -- hardware on-site.

All tests were carried out at EANTC's lab in Berlin, Germany with the exception of the orchestration campaign which was pre-staged and showcased in Austin, Texas.

Test Bed Topology
The test topology was fairly straightforward. NFVi vendors, with the support of testing solutions company Ixia and EANTC set up the hardware in EANTC's lab in Berlin. Each NFVi vendor provided its own hardware (for details see chart).

  • Cisco NFVi provided a full OpenStack deployment, including three redundant control nodes, three compute nodes, a redundant storage system, two build nodes and a router, plus a switch connecting the infrastructure internally until May 2016.


  • Starting in October, EANTC installed a Dell EMC NFVi. It is a full OpenStack environment, with three redundant control nodes, three compute nodes, three storage nodes, one platform manager, one control node and three switches (two leaf switches and a management switch). It also includes Dell EMC hardware element managers for deployment and management of the infrastructure.


  • Huawei provided FusionSphere with three hot-standby control nodes. The FusionSphere installation was upgraded in the summer and returned to EANTC with new hardware.


  • Juniper provided one Contrail server until April 2016 which implemented the OpenStack environment using the virtual deployment option: One virtual control node, one virtual compute node and a virtual router. This satisfied the minimum testing requirements in the first phase.


  • Nokia CloudBand provided a full OpenStack deployment for the whole year (three control and eight compute nodes including local storage, plus a router for internal connectivity).


  • ZTE provided Tulip Elastic Cloud System (TECS) from April onward. The TECS environment is a full OpenStack deployment using local storage to meet the NFVi testing requirements.

Overview of Results
It is great news that most VNF/NFVi test combinations passed our tests in the December 2016 campaign. These include VNFs from Fortinet, Infoblox, Mitel and Procera running on NFVis from Dell EMC, Huawei, Nokia and ZTE -- 11 successful test combinations. Three combinations failed because of ex-ante incompatibilities between OpenStack versions. This makes a success rate of 79% for our latest campaign.

Together with the earlier test campaign, 36 combinations passed in total and 17 failed or were not completed. The success rate across the year is 69%.

Figure 2:

Given the large number of potential test combinations and the limited testing time, EANTC did not aim for a full-mesh test campaign. The goal was to complete two test combinations per VNF, which worked out in some cases but not in others, often due to integration, configuration or troubleshooting delays. The solutions marked on the chart in blue were available until December 2016; the other vendors had not participated in the latest test campaign. Most cells left empty in the matrix above relate to test combinations that did not happen; very few relate to test combinations that failed. Fundamental interoperability issues are reported in the next section.

Individual Findings and Issues
Many of the findings described in this section are related to OpenStack -- kind of obvious as all participating vendors based their NFVi implementations on OpenStack. This is a great opportunity to align the industry’s nomenclature (all parties understand the term "heat template," for example) and to avoid reinventing the wheel (vendors equally benefit from OpenStack evolution).

At the same time, it is a burden -- different implementations are based on OpenStack but are not identical. Executive management and marketing may overlook this fact -- but it is very important in practice. We had to adapt our automation framework for each single test combination. Vendors supplemented the OpenStack releases used in our test bed (Juno, Kilo and Liberty) more or less extensively, resulting in interoperability problems.

These were mostly configuration problems that could be overcome during test preparation -- a big, positive difference to the first test campaign a year ago. As a result, test preparation took much longer than planned, in some cases literally weeks and weeks.

At the OpenStack Summit in Barcelona in October, delegates mostly talked about Newton or Ocata release planning. It's unlikely service providers will adopt, or even have access to, these new releases with their advanced features any time soon. The nature of vendor development cycles and network deployment processes will always result in lagging behind OpenStack releases by around two years. It might be wise to react with a different release and support model for OpenStack just for telecom service providers.

Here is a list of detailed findings areas:

OpenStack Configuration
EANTC created a single template for all NFVis to use. This helped us automate test execution and perform identical tests across multiple NFVis. The configurations were not always easy to implement -- we had to dig deep into the policy, tenants and topology configuration. Sometimes, issues are simple yet may result in time-consuming troubleshooting. Naming conventions were not always implemented correctly. Some NFVis did not like using underscore in their network names, for example.

In the end, we had to handle each NFVi configuration separately. The conclusion remains similar to previous phases: NFVi configuration consumes time. Vendor implementations differ too much to be handled consistently.

NFVi/VIM implementation bugs continued to bring us into trouble. We faced an issue where one of the vendor-supported NFVis lost database synchronization between its nodes which brought the whole NFVi to a halt. The NFVi had to be rebuilt from scratch. Luckily the NFVi vendor had a script to implement all configurations which meant little time was lost on recovering the NFVi. At the current maturity level, administrating and maintaining an OpenStack installation often requires directly interacting with the database.

OpenStack Security
We had no intentions to test security (yet) -- we just wanted everything to work smoothly in the closed environment of our lab. Vendor implementations of OpenStack security policies differ quite a bit, which often resulted in VNF data plane connectivity issues. VNFs that forwarded traffic on the Ethernet layer, i.e. were IP-transparent, were most affected.

IP and MAC protection means that traffic cannot simply traverse a layer 2 virtual machine. This traffic will be dropped by default in some OpenStack implementations unless a security rule is updated on the virtual ports. EANTC tuned the test automation code to handle the different cases.

OpenStack network policy improvements can potentially cause interruptions to existing VMs.

Under the hood, our security issues were all related to different Neutron configurations. There is no easy way for a user to tell which features are installed and enabled, how they are configured, and how this will affect VNF functionality. Neutron is kind of a special case since both its use in general as well as the individual features it provides are entirely optional. While the "hardcore" OpenStack community is aware of Neutron usage, some vendors seemed to struggle with Neutron's philosophy.

OpenStack Modules Version Consistency
NFVi implementations based on OpenStack are quite modular. When creating a distribution, vendors can pick and choose from a range of components that the OpenStack community works on in different projects. Common sense would advise to choose a consistent set of software modules from a single OpenStack release. In a telecom environment, this approach fails:

  • Service providers want to run multi-vendor NFV solutions. It is unfeasible to wait with upgrades -- if only, for security reasons -- until all vendors involved in a virtualized network solution provide an upgrade to the same version.


  • Service providers want to keep their NFV infrastructure up and running all the time. They must implement in-service-rolling updates instead of updating all parts of a system at once.

As a result, telecom operators require multi-version interoperability. Unfortunately, this cannot be taken for granted. In the latest test round, we came across three major issues related to multi-version inconsistencies:

  • One of the participating NFVis used two versions of the OpenStack Identity API, as implemented by Keystone, in parallel: Version 2 for the NFVi's command line interface (CLI), and version 3 for the graphical user interface (GUI). Although EANTC's automation scripts -- obviously connecting to the CLI -- were fine using KeyStone v2, we could not use them easily: Users had been created via the GUI before, with an extended feature set of the newer KeyStone version, so the v2 CLI denied any subsequent activities. It turned out later that the vendor supported v3 over the CLI as well -- this problem was again a configuration/transparency issue.

  • In another case, we used the Horizon component of OpenStack to modify IP subnetwork allocation ranges. When applied to OpenStack Liberty versions of Horizon, this action worked as expected. When we applied the same command to one of the NFVis based on OpenStack Kilo, Horizon still reported success -- but did not execute the requested action. It seems nobody had thought about a proper negative response for "reserved for future use" unsupported requests.

  • Some VNFs under test implemented special Heat resources. They required the installation of those resources on the target NFVi. This added on the efforts to evaluate the VNF and introduced an additional version incompatibility with the NFVi. Although this incompatibility is rather simple to detect, it will be responsible for many interoperability issues and disappointments in the future. We recommend all VNF vendors should clearly communicate the minimum requirements for the software environment.

These are just three examples. Obviously, software behavior changes across versions; clients often need to be flexible when working with different versions of services. Unfortunately, we could not easily figure the exact versions of services/components in an NFVi installation. Distributors (vendors) would need to provide such information out of band as of today. We strongly recommend to implement better API version control/status operations in OpenStack-derived NFV components.

Migrating VNFs: Not Really Trivial
Migrating all or some VNF resources from one resource zone to another reliably is a non-trivial task which will continue to require a lot of focus. Specifically, network functions consisting of multiple components (virtual machines aka VNFCs) and the migration across multiple data center sites can be complex, as well as the resource calculation for the target zone.

So far, we restricted our tests to migrating all compute resources of a VNF at once; most participating VNFs consisted of only a single VM anyway. We always migrated to a local target zone with sufficient resources.

But still we got into problems with VNFs that process traffic on the Ethernet layer, i.e. are not routers. When resources of such VNFs were migrated, the Ethernet (MAC) address tables of connected switches did not get updated. Although the VNF resources had moved, traffic was still sent to the old compute node.

Upgrading NFVis
Participating vendors told us that an upgrade of NFVis using older OpenStack versions (Icehouse; Kilo) is a complex process only possible at the vendor location. This is one of the reasons why we have run all tests with software versions that are not officially maintained by OpenStack anymore, or maintenance will be discontinued in a few months. Multiple development stages are typically recognized for telecom software upgrades:

  • Manual upgrade by vendor (return equipment)


  • Field-serviceable manual upgrade by vendor with major downtime


  • Field-serviceable manual customer upgrade with major downtime


  • Automated customer serviceable upgrade with low downtime


  • In-service software upgrade with zero downtime.

Service providers want to get to the final stage quickly. The industry is currently at the first step for the OpenStack versions we have seen in our lab, and mentioned they plan to advance to the second step soon.

VNF Manager Interoperability Tests

We decided not to publish results of the VNFM/VIM interoperability test campaign. There were too many issues that might not be representative for the industry, and we would not have been able to reasonably anonymize our reporting.

We will log the lessons learned and rerun the test campaign with newer software versions in the future.

Summary
A year ago, when we ran the first test campaign, many findings were new and it was difficult to overcome them. Vendors supplied new software in some cases; in 11 cases, we had to give up the test combination as not achievable. In the latest campaign, VNF and NFVi vendors generally understood the challenges better and provided more advanced software; most of the issues were related to configuration. Only three combinations failed because of major incompatibilities.

Figure 3: Hotbed of Activity Test bed where EANTC put participating NIA member products through their interoperability paces. Test bed where EANTC put participating NIA member products through their interoperability paces.

Still, the amount of prestaging time, the number of configuration questions, and the complexity of issues to overcome in each individual multi-vendor combination is very difficult to predict. Seemingly very experienced vendors can struggle with a tiny software engineering issue for days and weeks. In quite a few cases, the configuration issues were not even related to the multivendor aspect: They often had to do with basic misconfigurations of their own solution. Even though there were fewer "insurmountable" problems seen in the latest test campaign, the unpredictability is still a major issue.

Our dream would be greatly streamlined open source solutions with reduced number of options, improved installation/configuration procedures and more extensive telecom focused baseline software in OpenStack. We hope that the industry might understand open source does not substitute standards at all, and will develop normative APIs very soon. In addition, much better software documentation and more readily available education programs for support engineers would be high on our wish list.

Outlook
The journey continues: It is noticeable that vendors have kept up the pace of development and have a lot of plans for management and orchestration support in 2017. The NIA Board has approved a test campaign covering NFV orchestration and VNF manager to VIM interoperability for the first quarter of 2017. It will be followed by a multi-vendor SDN and NFV integration test in the second quarter.

The New IP Agency intentionally does not have a technical committee developing test plans. We plan to align the Q1 test session with ETSI's draft interop test specification TST007, being developed by EANTC and other ETSI members right now. Any test plan insights will be fed back to ETSI.

In parallel, we will continue to further automate the existing test programs, most specifically the VNF to NFVi interoperability effort. The test plans will be amended by requests from service provider members. Our goal is to continually expand the tests with realistic cases, reducing service provider effort for baselining the interoperability status of solutions they intend to purchase.

— Carsten Rossenhövel is Managing Director of the European Advanced Networking Test Center (EANTC), which conducted the tests for the New IP Agency. Special to The New IP

CALLING ALL TEST, ASSURANCE AND MONITORING COMPANIES:
Make sure your company and services are listed free of charge at Testapedia, the comprehensive set of searchable databases covering the companies, products, industry organizations and people that are directly involved in defining and shaping the telecom test and measurement industry.

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like