EANTC & Cisco: A New Era of Testing

Industry test specialists at EANTC explain why their visit to Cisco's labs to test some of the vendor's service provider virtualization solutions opened up a new chapter for them and the market.

May 6, 2015

6 Min Read
EANTC & Cisco: A New Era of Testing

Communications service providers are understandably excited at the prospect of being able to develop and provision services more quickly, easily and efficiently using SDN, NFV and a new generation of network management tools. But there are a lot of questions to be answered about how new virtualization capabilities work and how they can be tested.

In an effort to answer some of these questions, a team from EANTC, supported by Light Reading, flew west to Cisco's labs in San Jose, Calif., to conduct a series of validation and verification exercises on some of the vendor's cloud, SDN and virtualization platforms.

The result was a mammoth report, published here on Light Reading in March, that explained what the European Advanced Networking Test Center AG (EANTC) team did and their findings and conclusions. (See Validating Cisco's Service Provider Virtualization & Cloud Portfolio.)

The visit to Cisco Systems Inc. (Nasdaq: CSCO) and the tests broke new ground: Such tests had not been done before.


So what did the EANTC team need to do to conduct these tests? What did they learn? And what's next? We talked to Jambi I. Ganbar, senior technical marketing manager, and Carsten Rossenhövel, managing director at EANTC, to find out.

Light Reading: What was new about the test methods you used for the Cisco service provider virtualization and cloud system evaluation?

EANTC: This was really different to traditional tests.

Traditionally, we have based our performance tests on methodologies standardized by the IETF or from our own test case database built up over many years. In these cases, we are testing a function on a router, such as an In Service Software Upgrade (ISSU).

The Cisco validation was quite different because it was about verifying provisioning and network management processes and validating virtual service functionality -- can a service be delivered and does it function as advertised? This was very different from testing the scalability, QoS and failure conditions of hardware platforms.

Light Reading: Were you able to plan much in advance?

EANTC: Very little. Cisco disclosed the full test scope only when we arrived on site, so it was pretty much all happening at the scene. Because this was the first test of its kind we had to build a plan and a methodology as we went along. For example, what is the test goal for an evaluation of the NSO [network service orchestrator]? We felt it was important to validate service configuration in comparison with traditional methods: We had to figure out whether NSO was doing it more quickly or more efficiently.

So, for example, we were able to see that when I typed in 15 lines of NSO configuration data, it generated 318 lines of Cisco configuration that made sense -- the resulting lines were not random. And we could see that everything was in place and configured correctly. (See Cisco Network Service Orchestrator – Service Provisioning.)

The use of NSO to push down configuration to the routers from a GUI tool enables the network engineering group at a service provider to simplify their operations and reduce the amount of potential errors. In addition, it also helps to bridge the gap between different network operating systems and their various generations. The operator no longer needs to know the syntax to configure a Juniper LSP [label-switched path] and a Cisco LSP. Understanding the logic is, nonetheless, still essential and topics such as Traffic Engineering are taking a front row seat in such topologies.

Light Reading: What was the reaction from the Cisco team to your tests?

EANTC: The spirit was very collaborative -- the Cisco team was very supportive. They know how it functions and they know what concerns service providers. The EANTC team explored the broad functionality as we would do in a service provider proof-of-concept (PoC) test. According to the Cisco team, we were asking the same questions that their service provider customers have been asking, so that was encouraging.

Light Reading: Were you surprised at the outcome/results of the test?

EANTC: Well, Cisco was letting us into their labs, so we guessed they would be pretty confident it was going to work, right? That said, the incredible level of integration and the coherent vision that Cisco has developed from ground up was surprising in a positive way. If you look at the individual elements -- the WAE [WAN Automation Engine], the NSO, the dCloud -- we expected them to work. But the fact that we were able to confirm Cisco's claims in the first test session, without hiccups, was quite impressive.

In one of the Cloud VPN tests, when our team acted like a small business customer -- we called it Lakshmi's Café -- and wanted to order a cloud VPN, all Cisco hardware components, virtual functions and management systems worked together as expected. It was like shopping at Amazon.com. The entire system worked together. This really brought the story home -- it is very hard for any vendor to integrate such a lot of component parts. (See The Cloud Services Creation and Delivery Stage.)

Light Reading: Do the tools exist for you to be able to conduct all the tests?

EANTC: A lot of functional testing aspects -- as we conducted in this project -- can be evaluated without the need for advanced test tools. But as soon as we get into the scalability, performance and quality of experience aspects of NFV, new test solutions need to be developed. EANTC actively contributes to the ETSI NFV ISG's Testing Working Group (TST) to develop interoperability and performance test methodologies.

Light Reading: What's next for EANTC in terms of virtualization/cloud testing?

EANTC: We have looked at orchestration, virtualization and SDN and will continue to undertake cloud and virtualization tests. We are looking to test additional solutions from multiple vendors, which will enable us to test other approaches -- Cisco has a very specific approach to service provider virtualization. Other vendors have other approaches, use other protocols and have other ways of engaging and working with customers. That would provide additional value, getting different viewpoints. In the future we will look to test best-of-breed solutions, which is what service providers will want but which they cannot get just now. That would involve testing integrated solutions from multiple vendors.

— Ray Le Maistre, Circle me on Google+ Follow me on TwitterVisit my LinkedIn profile, Editor-in-Chief, Light Reading

Read more about:

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like