2016 NIA-EANTC NFV Interoperability Test Report
In the first year of its industry-wide interoperability program, the New IP Agency included 33 commercial solutions from 23 member companies. We conducted three test campaigns focusing on virtualized network functions (VNFs), the NFV infrastructure, VNF managers and NFV orchestrators. The tests identified more than eight general areas of interoperability issues seen today; they helped vendors to improve their solutions and resolve problems in multivendor scenarios not tested before. Vendor members have upstreamed test findings to open source projects such as OpenStack, and to standards-developing organizations such as ETSI.
NIA's place is to test multivendor interoperability industry-wide, independent of any specific commercial or non-commercial ecosystems, to ensure service providers can choose their implementations from the widest selection of offerings.
The number of NIA testing program participants has expanded over the year, and the success rates have improved gradually. This is great news -- but let's not get the wrong message: This industry is still in its early stages, and no success can be taken for granted unless tested individually.
The New IP Agency philosophy is to conduct established test campaigns pertinent to communication service providers' adoption and deployment of NFV solutions to broaden participation and improve interoperability in well-understood NFV areas such as basic life-cycle management.
Simultaneously, we aim to innovate with new test campaigns, accelerating interoperability of new developments and being among the first to validate as-yet unproven grounds. In both cases, we aim to conduct the most realistic, in-depth, and transparently documented public interoperability tests worldwide.
To this end, we have run our VNF-to-NFVi interoperability program throughout the course of 2016. Additionally, we complemented the VNF-to-NFVi evaluation with a one-time Service Chaining Orchestration interoperability showcase, hosted in June, and an initial VNF Manager-to-VIM interoperability campaign. Let's have a look at each of the three:
VNF-to-NFVi Interoperability Program
This first NIA testing program covers the basics: Image management and life-cycle management. We verify whether one party's VNF can be instantiated, configured, and terminated on an NFVi of another party. This may sound trivial; after some 50+ test combinations, we can assure the reader it is not. In a nutshell, interoperability issues show up because OpenStack implementations vary a lot, and most vendors have not (or not seriously) tested with each other before. (We cover the types of issues and their root causes further below.)
The tests include graceful and forceful VNF restart, scaling up (by increasing CPU capacity), and modifying network resources during operation.
VNF Manager-to-VIM Interoperability Campaign
The second NIA testing program began with a single campaign this year. It focuses on the interoperability between VNF managers and the virtual infrastructure management (VIM). The VNF manager's task is to manage all VNF compute, network and storage resources. VNF managers are specifically helpful to the deployment of complex VNFs consisting of multiple virtual machines (VMs) which are required to scale out and in, adding and removing resource instances such as VMs.
Our tests cover life-cycle management functions on the VNF level that the VNFM exposes to the northbound management and orchestration. In addition, we suggest resource migration, resource management, fault management and performance monitoring test cases.
For 2016, we decided not to include, or at least not focus on, the role of the NFV Orchestrator (NFVO) in this test program. The intent being to focus on VNFM/VIM interoperability tests as purely as possible. In addition, we aimed to focus on individual, VNF-specific VNFMs: These could potentially take scale decisions based on individual VNF (application-layer) conditions, and generally include element manager (EM) functions.
Service Chaining Orchestration Showcase
In June this year, NIA conducted a one-time, on-site interoperability test and showcase. The goal was to construct a complete NFV scenario including NFVi, VIM, VNFs and NFVO and showcase service chaining in a short time. We chose a representative virtualized CPE (vCPE) use case. The test plan included basic vCPE service creation, static and elastic bandwidth provisioning and service chaining with generic and flow-based forwarding graphs.
All NIA vendor members were eligible to participate, and 23 did -- equivalent to 90%. This year, a total of seven NFVis, three NFVOs and 23 VNFs participated.
Many vendors supported a subset of test campaigns, depending on their available resources, software readiness and -- in the case of NFVis -- hardware on-site.
All tests were carried out at EANTC's lab in Berlin, Germany with the exception of the orchestration campaign which was pre-staged and showcased in Austin, Texas.
Test Bed Topology
The test topology was fairly straightforward. NFVi vendors, with the support of testing solutions company Ixia and EANTC set up the hardware in EANTC's lab in Berlin. Each NFVi vendor provided its own hardware (for details see chart).
Overview of Results
It is great news that most VNF/NFVi test combinations passed our tests in the December 2016 campaign. These include VNFs from Fortinet, Infoblox, Mitel and Procera running on NFVis from Dell EMC, Huawei, Nokia and ZTE -- 11 successful test combinations. Three combinations failed because of ex-ante incompatibilities between OpenStack versions. This makes a success rate of 79% for our latest campaign.
Together with the earlier test campaign, 36 combinations passed in total and 17 failed or were not completed. The success rate across the year is 69%.
Given the large number of potential test combinations and the limited testing time, EANTC did not aim for a full-mesh test campaign. The goal was to complete two test combinations per VNF, which worked out in some cases but not in others, often due to integration, configuration or troubleshooting delays. The solutions marked on the chart in blue were available until December 2016; the other vendors had not participated in the latest test campaign. Most cells left empty in the matrix above relate to test combinations that did not happen; very few relate to test combinations that failed. Fundamental interoperability issues are reported in the next section.
Individual Findings and Issues
Many of the findings described in this section are related to OpenStack -- kind of obvious as all participating vendors based their NFVi implementations on OpenStack. This is a great opportunity to align the industry’s nomenclature (all parties understand the term "heat template," for example) and to avoid reinventing the wheel (vendors equally benefit from OpenStack evolution).
At the same time, it is a burden -- different implementations are based on OpenStack but are not identical. Executive management and marketing may overlook this fact -- but it is very important in practice. We had to adapt our automation framework for each single test combination. Vendors supplemented the OpenStack releases used in our test bed (Juno, Kilo and Liberty) more or less extensively, resulting in interoperability problems.
These were mostly configuration problems that could be overcome during test preparation -- a big, positive difference to the first test campaign a year ago. As a result, test preparation took much longer than planned, in some cases literally weeks and weeks.
At the OpenStack Summit in Barcelona in October, delegates mostly talked about Newton or Ocata release planning. It's unlikely service providers will adopt, or even have access to, these new releases with their advanced features any time soon. The nature of vendor development cycles and network deployment processes will always result in lagging behind OpenStack releases by around two years. It might be wise to react with a different release and support model for OpenStack just for telecom service providers.
Here is a list of detailed findings areas:
EANTC created a single template for all NFVis to use. This helped us automate test execution and perform identical tests across multiple NFVis. The configurations were not always easy to implement -- we had to dig deep into the policy, tenants and topology configuration. Sometimes, issues are simple yet may result in time-consuming troubleshooting. Naming conventions were not always implemented correctly. Some NFVis did not like using underscore in their network names, for example.
In the end, we had to handle each NFVi configuration separately. The conclusion remains similar to previous phases: NFVi configuration consumes time. Vendor implementations differ too much to be handled consistently.
NFVi/VIM implementation bugs continued to bring us into trouble. We faced an issue where one of the vendor-supported NFVis lost database synchronization between its nodes which brought the whole NFVi to a halt. The NFVi had to be rebuilt from scratch. Luckily the NFVi vendor had a script to implement all configurations which meant little time was lost on recovering the NFVi. At the current maturity level, administrating and maintaining an OpenStack installation often requires directly interacting with the database.
We had no intentions to test security (yet) -- we just wanted everything to work smoothly in the closed environment of our lab. Vendor implementations of OpenStack security policies differ quite a bit, which often resulted in VNF data plane connectivity issues. VNFs that forwarded traffic on the Ethernet layer, i.e. were IP-transparent, were most affected.
IP and MAC protection means that traffic cannot simply traverse a layer 2 virtual machine. This traffic will be dropped by default in some OpenStack implementations unless a security rule is updated on the virtual ports. EANTC tuned the test automation code to handle the different cases.
OpenStack network policy improvements can potentially cause interruptions to existing VMs.
Under the hood, our security issues were all related to different Neutron configurations. There is no easy way for a user to tell which features are installed and enabled, how they are configured, and how this will affect VNF functionality. Neutron is kind of a special case since both its use in general as well as the individual features it provides are entirely optional. While the "hardcore" OpenStack community is aware of Neutron usage, some vendors seemed to struggle with Neutron's philosophy.
OpenStack Modules Version Consistency
NFVi implementations based on OpenStack are quite modular. When creating a distribution, vendors can pick and choose from a range of components that the OpenStack community works on in different projects. Common sense would advise to choose a consistent set of software modules from a single OpenStack release. In a telecom environment, this approach fails:
As a result, telecom operators require multi-version interoperability. Unfortunately, this cannot be taken for granted. In the latest test round, we came across three major issues related to multi-version inconsistencies:
These are just three examples. Obviously, software behavior changes across versions; clients often need to be flexible when working with different versions of services. Unfortunately, we could not easily figure the exact versions of services/components in an NFVi installation. Distributors (vendors) would need to provide such information out of band as of today. We strongly recommend to implement better API version control/status operations in OpenStack-derived NFV components.
Migrating VNFs: Not Really Trivial
Migrating all or some VNF resources from one resource zone to another reliably is a non-trivial task which will continue to require a lot of focus. Specifically, network functions consisting of multiple components (virtual machines aka VNFCs) and the migration across multiple data center sites can be complex, as well as the resource calculation for the target zone.
So far, we restricted our tests to migrating all compute resources of a VNF at once; most participating VNFs consisted of only a single VM anyway. We always migrated to a local target zone with sufficient resources.
But still we got into problems with VNFs that process traffic on the Ethernet layer, i.e. are not routers. When resources of such VNFs were migrated, the Ethernet (MAC) address tables of connected switches did not get updated. Although the VNF resources had moved, traffic was still sent to the old compute node.
Participating vendors told us that an upgrade of NFVis using older OpenStack versions (Icehouse; Kilo) is a complex process only possible at the vendor location. This is one of the reasons why we have run all tests with software versions that are not officially maintained by OpenStack anymore, or maintenance will be discontinued in a few months. Multiple development stages are typically recognized for telecom software upgrades:
Service providers want to get to the final stage quickly. The industry is currently at the first step for the OpenStack versions we have seen in our lab, and mentioned they plan to advance to the second step soon.
VNF Manager Interoperability Tests
We decided not to publish results of the VNFM/VIM interoperability test campaign. There were too many issues that might not be representative for the industry, and we would not have been able to reasonably anonymize our reporting.
We will log the lessons learned and rerun the test campaign with newer software versions in the future.
A year ago, when we ran the first test campaign, many findings were new and it was difficult to overcome them. Vendors supplied new software in some cases; in 11 cases, we had to give up the test combination as not achievable. In the latest campaign, VNF and NFVi vendors generally understood the challenges better and provided more advanced software; most of the issues were related to configuration. Only three combinations failed because of major incompatibilities.
Still, the amount of prestaging time, the number of configuration questions, and the complexity of issues to overcome in each individual multi-vendor combination is very difficult to predict. Seemingly very experienced vendors can struggle with a tiny software engineering issue for days and weeks. In quite a few cases, the configuration issues were not even related to the multivendor aspect: They often had to do with basic misconfigurations of their own solution. Even though there were fewer "insurmountable" problems seen in the latest test campaign, the unpredictability is still a major issue.
Our dream would be greatly streamlined open source solutions with reduced number of options, improved installation/configuration procedures and more extensive telecom focused baseline software in OpenStack. We hope that the industry might understand open source does not substitute standards at all, and will develop normative APIs very soon. In addition, much better software documentation and more readily available education programs for support engineers would be high on our wish list.
The journey continues: It is noticeable that vendors have kept up the pace of development and have a lot of plans for management and orchestration support in 2017. The NIA Board has approved a test campaign covering NFV orchestration and VNF manager to VIM interoperability for the first quarter of 2017. It will be followed by a multi-vendor SDN and NFV integration test in the second quarter.
The New IP Agency intentionally does not have a technical committee developing test plans. We plan to align the Q1 test session with ETSI's draft interop test specification TST007, being developed by EANTC and other ETSI members right now. Any test plan insights will be fed back to ETSI.
In parallel, we will continue to further automate the existing test programs, most specifically the VNF to NFVi interoperability effort. The test plans will be amended by requests from service provider members. Our goal is to continually expand the tests with realistic cases, reducing service provider effort for baselining the interoperability status of solutions they intend to purchase.
— Carsten Rossenhövel is Managing Director of the European Advanced Networking Test Center (EANTC), which conducted the tests for the New IP Agency. Special to The New IP
CALLING ALL TEST, ASSURANCE AND MONITORING COMPANIES: Make sure your company and services are listed free of charge at Testapedia, the comprehensive set of searchable databases covering the companies, products, industry organizations and people that are directly involved in defining and shaping the telecom test and measurement industry.