Validating Cisco's Service Provider Virtualization & Cloud PortfolioValidating Cisco's Service Provider Virtualization & Cloud Portfolio

How can CSPs develop and offer new services with the same speed and flexibility as web services/OTT players such as Google, Amazon and Netflix?

During the course of the verification process we heard Cisco refer to urgency, agility and time-to-deployment like we never have before: That sort of talk is normally the domain of software development teams.

Cisco explained that its service provider customers are at different stages of evaluating, deploying and leveraging their network functions virtualization (NFV), software-defined networking (SDN) and cloud-based applications offerings.

For this reason, Cisco provided EANTC with access to various tools, labs resources and other capabilities to, in essence, play the role of a CSP that is offering business VPN services to its customers.

Cisco's goal was to take us on a three-stage journey very similar to that being taken by many CSP customers that are creating and delivering business VPN services.

Step 1: The CSP Evaluation and Proof of Concept Stage
CSPs can begin their journey by experimenting with web portals and using a "cloud in a box" tool that helps them get to grips with the possibilities of programming a virtual, elastic environment.

Step 2: The Deployment Stage: Cisco Service Provider SDN Deployment Tools
Once such experiments are over and CSPs understand the possibilities, Cisco can then support their customers with development tools that will help them create the cloud solution that best suits them, as well as products that enable CSPs to deploy network capacity and services.

Step 3: The Cloud Services Creation and Delivery Stage
Cisco offers the capabilities to manage and run network services and applications using cloud-enabled data centers. This is a long-term play -- a cloud service based on Cisco's expertise in networking and the data center. To illuminate the possibilities offered by this service, Cisco prepared some applications that demonstrated how CSPs can generate revenues from their cloud platform investments.

First up -- Step 1.

Next page: Step 1: The CSP Evaluation and Proof of Concept Stage

To go back to the introduction, please click on this link.

Step 1: The CSP Evaluation and Proof of Concept Stage
As a result of EANTC's engagement with ETSI's NFV Industry Specifications Group, which is run by a core group of major telecom operators, we know that the largest CSPs in the world are actively working on accelerating the adoption of virtual network functions, developing cloud services and adopting web-scale operations.

There are hundreds of other network operators too, of course, and many (if not all) will be seeking help on how to make their networks and operations more efficient and flexible. So how can CSPs of all types, no matter what their resources, engage with Cisco and figure out what the vendor has to offer in terms of cloud enablement and virtualization?

One entry-level resource is a web-based platform called dCloud (http://dcloud.cisco.com/), an online resource that is freely available for anyone to access and use and which, in essence, illustrates what is possible in the New IP world. To validate this tool, an EANTC engineer created an account and ran the scenarios himself. Once an account is created, the user is given a set of pre-defined setups that demonstrate some of Cisco's newer offerings and collaborations, the latter of which includes an SDN controller based on OpenDayLight specifications. In total, we encountered 40 different scenarios. We ran through three different dCloud scenarios, each of which was a preview of a real application that we tested later in the two-week process.

The scenarios were:

Cisco WAN Automation Engine 6.0 with 8-Nodes v1

Cisco CloudVPN Technology Preview v2

Cisco Network Control Systems 3.3 MPLS VPN v1

In this phase of the verification, dCloud was used to provide us with a preview of what we would encounter later. As part of our processes, we requested that one of the dCloud scenarios (called Sandbox) be connected to our own laptops for independent verification. This was easy to enable and we were able to run the scenario via an external connection from the Cisco Sandbox to our laptop device.

Cisco's dCloud is an enabler designed for two types of user: Cisco sales engineers and account teams; and customers. It can connect to public clouds, such as Amazon Web Services (AWS), as well as to a CSP's own network infrastructure.

If a CSP prefers not to use a public cloud for the evaluation process, maybe because of concerns related to the security of its systems and data, Cisco has a solution -- an innovation and development pod called /dev/innovate.

This innovation pod is a self-contained data center rack that Cisco leases to CSPs on an annual basis. The pod includes a range of Cisco products that a CSP can use to build cloud services:

The pod also has enough rack space to host any additional components, enabling the CSP to incorporate additional systems from Cisco or even from other vendors.

The aim of the pod is to enable a CSP's R&D team to directly interact with that CSP's operations group in order to address questions around scalability, security, high availability, and so on. It has other benefits too, as it could shorten the time needed to get from concept to service development and creation and bring CSPs closer to a "web speed" development timeframe.

All of this is part of Cisco's efforts to help CSPs speed up the decision-making and development processes as they consider their virtualization strategies.

The next logical step, following the evaluation stage, is for a CSP to move to the deployment process.

Next page: Step 2: The Deployment Stage – Cisco Service Provider SDN Deployment Tools

To go back to the introduction, please click on this link.

Step 2: The Deployment Stage – Cisco Service Provider SDN Deployment Tools
For the deployment of an SDN-enabled service provider cloud, Cisco has developed its Evolved Services Platform (ESP). This is designed to provide an agile, automated and reliable operation environment that, Cisco states, includes predictable design, capacity planning and multi-vendor provisioning.

The two key elements of ESP that Cisco provided for the test are:

the Cisco WAN Automation Engine (WAE), developed following the acquisition of Cariden in late 2012;

the Cisco Network Service Orchestrator enabled by Tail-f, which, as the name suggests, has been developed from technology acquired as part of the purchase of Tail-f in 2014.

Cisco's WAN Automation Engine provides a lot of very useful functionality for a CSP. The tool collects historical network utilization data and is able to predict future behavior based on past and present heuristics. In essence, the software is used to manage traffic in CSP backbone networks. Managing backbone traffic is typically a question of understanding congestion and congestion-related events, a powerful concept that ties very well to the fundamentals of SDN.

WAE uses Path Computation Element Protocol (PCEP), as well as BGP-LS, to learn the network topology and install state in network elements. To show how it works, Cisco set up a large demonstration network comprising 35 routers, including the ASR9006, CRS-4/S and several models of the Cisco 12000 product family. As it is rare that any network operator would deploy infrastructure from a single supplier, Cisco also incorporated IP infrastructure from a third-party vendor.

Judging from EANTC's 12-year history of annual multi-vendor interoperability testing at the MPLS World Congress in Paris, this was a very positive and meaningful decision.

Armed with the details of what was to be used in the verification process, we headed to the lab.

Our first task was to disconnect a link between two of the devices and examine the outcome so that we could be sure that what we were seeing was a real network setup.

Figure 1: Figure 1: Cisco MATE Live Network Overview (part of Cisco WAE)

The tests we executed with WAE were designed to verify its functionality as well as demonstrate that web-based applications could be installed on top of WAE. To achieve, the verification process was split into two sections: A CSP's offline activities; and live network operations activities.

Next page: WAN Automation Engine: Maintenance Planning

To go back to the introduction, please click on this link.

WAN Automation Engine: Maintenance Planning
In our view, having a system that informs a network operator about the effect that device maintenance will have on availability and performance is beneficial. In this functional test, we prepared for a maintenance activity on the interfaces of one of the routers.

We used MATE Live, which is part of the WAE tool suite, to indicate that we intend to take an interface (in this case, on a node called RP2) offline at a certain time. As part of that process, we also factored in the possibility that a network failure might occur while the RP2 maintenance was underway and requested the application to include that in its report.

With those requests submitted, the system reported back with a long list of consequences resulting from the combination of the maintenance and additional node failure. In bold green letters the MATE Live reported "PASS – no congestion requirements violated."

We received a detailed list of the resulting connections to alternative devices, with the software predicting that the link utilization on the following routes would be up to 98.7% but would not exceed 100% in any instance.

With those details as our guide, we decided to put the network, and the MATE Live software's predictions, to the test. The Cisco team informed us that initiating the procedure would generate a "paper trail" that would show whether MATE Live provided an accurate prediction.

So while test traffic was running in the network, we entered the lab and took down the identified RP2 interface to disturb the network's topology. We then compared the bandwidth predicted by MATE Live with the actual bandwidth and found them to be almost identical (591.06 Mbit/s predicted compared with 590.02 Mbit/s recorded by EANTC -- the application was only 1.04 Mbit/s out).

Next page: Cisco WAN Automation Engine – Disjoint Tunnel Creation

To go back to the introduction, please click on this link.

Cisco WAN Automation Engine – Disjoint Tunnel Creation
For a few mission-critical applications, CSP networks are sometimes configured to transmit two copies of the same traffic streams for extra protection against packet loss. As the Cisco team noted, this is particularly important when the content is video and the content provider does not want to lose any packets in case failure occurs.

So, how can a CSP guarantee that two transport paths are diverse enough so that the content provider is not risking the loss of any packets in the case of a single link or node failure?

In essence the WAE software function was clear: Verify that two MPLS transport tunnels, entering the network at different locations but exiting the network at the same location, do not share any link or node in the path.

To verify that this function works, we used the MATE tool to build a pair of LSP (label-switched path) tunnels between the R30 and R35 nodes (the latter node being a third-party router, not a Cisco device). We generated test traffic and verified via device CLI (command line interface) that the traffic was passing over the two tunnels.

Figure 2: Figure 2: Dual Tunnel Setup using Cisco MATE Live

Next we used the tunnel disjoint operation to split the Label Switched Paths (LSPs) we just built.

Figure 3: Figure 3: Disjoint LSP operation using Cisco MATE Live

Once the tunnel disjoint command ran, we logged back into the routers to find a very different configuration than the simple one line that was used to create the LSPs previously.

We then waited five minutes and verified that the links were being used based on the Ixia N2X tester traffic streams we configured in the network. We checked to see that the configuration was installed on both Cisco and the third party router. At that point we increased the rate of the test traffic and monitored that the colors of the lines changed based on utilization (the tool uses colors to symbolize link utilization that made for a nice visual effect over time).

Figure 4: Figure 4: The resulting disjointed LSPs displayed by Cisco MATE Live.

We were interested in the mechanics and asked how MATE Live actually created a configuration on the routers. Cisco explained that the system interacts with the Cisco Network Service Orchestrator enabled by Tail-f using a standards-based API. MATE Live uses a model-processing tool that sends instructions to the Network Service Orchestrator (NSO), which in turn connects to the devices.

The function that MATE Live provides is the equivalent of a "what if" function. Once the CSP is confident that a change should take place and confirms using the MATE Live GUI, NSO kicks in and sends the configuration changes to the routers.

We noted, by comparing between pre- and post-configuration data that the post-tunnel disjoint operation was significantly more complicated -- in terms of configuration it's easier for the system to set up a path than to tear it down. But the CSP -- in this instance, our EANTC test engineer -- did not actually have to spend any time creating the device configuration.

But before we dive deeper into NSO, we still had other WAE functions we needed to verify.

Next page: Cisco WAN Automation Engine – Bandwidth On Demand

To go back to the introduction, please click on this link.

Cisco WAN Automation Engine – Bandwidth On Demand
The next verification process involved a set of web-based applications that run on WAE.

What is the relationship between a web browser application and the WAE software? As explained on Cisco's DevNet, WAE exposes REST (Representational State Transfer) application programming interfaces (APIs) that enable developers (in this instance from Cisco) to develop applications that can run in a web browser.

We used the web browser's developer view to capture the interaction between the browser and the application and confirmed that REST communications with WAE were being made.

The first application we tested was Bandwidth on Demand. This tool allows the CSP to dynamically add service capacity between network nodes, even, as Cisco explained, for just a short period of time.

We immediately identified the benefit to end users of this capability since we occasionally host large groups of vendor engineers during EANTC's public interoperability events, which last for a few weeks. During this time, our network connection becomes congested and, as a consequence, it slows down considerably. If we could use a web tool to add bandwidth for those weeks -- and only for those weeks -- all these engineers we are hosting will enjoy faster download times and will have an easier time collaborating with their developer colleagues who are often located in faraway locations.

To test this functionality, we created a service (in this case, an MPLS Label Switch Path), between two nodes (R35 and R30) and sent traffic at 762 Mbit/s. Since the link capacity in the lab was 1 Gbit/s for the majority (but not all) of the path, the GUI displayed the link utilization in yellow for the GigE links and green for the 10GigE link.

The colors signify certain levels of network utilization and are used to warn CSPs of the likelihood of congestion: The CSP can set the thresholds, with green representing utilization below a certain level and yellow above a certain level. The system can also display links in red, which signifies that an even higher threshold has been breached.

Figure 5: Figure 5: Service Path Utilization in the Network

We then used the bandwidth on demand tool to request an additional 400 Mbit/s for our service. The application provided the report as shown in the screenshot below. As you can see, the system reported congestion on the first hop in our path (other traffic was running in the network as one would expect in a CSP network).

Figure 6: Figure 6: Bandwidth on Demand Utilization Report

We also received an option that prompted us to "optimize". Once we selected this option, the system recommended a new path (as displayed below in Figure 7).

Since we did not specify any hop limit or specific latency, we were curious to see how the application intended to provision additional capacity for our service. The solution was a network route that most likely would not have been selected by a human engineer, with the brown arrows signifying the "what if" routes suggested by the tool:

Figure 7: Figure 7: Bandwidth on Demand Recommended Path

As Figure 7 shows, the system created a complete new path that actually took a longer route from A to Z and arrived at the end point (marked by Z on the topology) from an unpredicted direction (to our eyes, it arrived from the "rear").

The number of nodes in our path changed from 5 to 7, but by using this path, both LSPs, as the colors indicate, predicted less than 50% utilization. Once we accepted the configuration, we generated traffic again, this time at the cumulative rate (762 Mbit/s plus 400 Mbit/s) and saw the links on the topology change their colors to light green, an indication that no link was congested. We also recorded zero traffic loss, which was what we expected.

As one of the Cisco managers who attended the process noted, "this is a little bit like an autopilot on a plane -- you still need a pilot that really knows what he is doing on the plane to operate it."

This indeed is a novel approach to network engineering, where the system is responsible for developing solutions to problems and the network operator is merely supervising the system.

Next page: Cisco WAN Automation Engine – Bandwidth Calendaring

To go back to the introduction, please click on this link.

Cisco WAN Automation Engine – Bandwidth Calendaring
If an application has an extensive database of past and present network states -- one that includes link utilization, latency, and device layout -- the application could attempt to calculate (or "predict") future utilization of the network.

The next application Cisco demonstrated for this purpose is called Bandwidth Calendaring. This application enables a service provider's customer to reserve bandwidth for a limited period of time in the future.

EANTC played the role of the enterprise in this process by using a web interface to order more bandwidth. Such a capability is a great way to facilitate speedy off-site backups and database synchronization -- enterprise IT activities that happen infrequently and require a lot of bandwidth for a short period of time.

In order to really verify that the system was functioning as expected, we started a 100Mbit/s test stream across the service. Since our initial service parameter included just 10 Mbit/s, we recorded 90Mbit/s traffic loss. We then used the Bandwidth Calendaring web GUI to increase our capacity to 100 Mbit/s for a duration of 10 minutes.

We clicked the "go" button and almost immediately the traffic generator stopped reporting traffic loss.

This speedy reaction from the network, something that an enterprise customer could immediately use, is a great way for service providers to improve their service offerings and revenue streams.

There is another element to this tool that is worth noting. As the web interface shows how this application will run in an international telecom network, and not in confined test lab, we noted that the nodes were geographically spread on a map and that latency was an element that could be configured when making the bandwidth calendaring request.

Figure 8: Figure 8: Cisco's WAN Automation Engine Bandwidth Calendaring Application Interface

Cisco explained that latency could be taken from geographical locations, based on latitude and longitude parameters, as well as by using the Cisco Prime network management tool (which was not part of the test) to monitor the network and actually add measurement data to WAE's database.

Since we did not have an impairment generator in the lab, which would have allowed us to simulate different link latencies, we could not verify latency constraints would have worked, but we expect this evaluation to be part of a follow-up test.

Next page: Cisco WAN Automation Engine – Tunnel Split/Merge Manager

To go back to the introduction, please click on this link.

Cisco WAN Automation Engine – Tunnel Split/Merge Manager
The final web application running over WAE that we verified is an optimization tool called Tunnel Split/Merge.

The prime use case is a brownfield deployment where a large number of Label Switched Paths (LSPs) already exist in the network. The idea is that once WAE is introduced into the network, the service provider could re-optimize the LSPs, removing those no longer needed and optimizing others.

Another use case is where automatic bandwidth LSPs are used in the network. In this instance, Cisco is trying to address the problem faced when the bandwidth setup on the LSPs becomes so great that, in the event of failure, the network is unable to signal for backup links.

Whatever the use case, being able to optimize the network and essentially grab back wasted network resource is beneficial in bandwidth-constrained networks.

To verify that the tool functions as it should, we specified maximum and minimum bandwidth thresholds for all LSPs in the test network. Cisco WAE queried the network and returned a list of recommendations about which LSPs to split and which LSPs to merge. We clicked "Select all" and then the "Optimize" button.

Figure 9: Figure 9: Cisco's WAN Automation Engine Tunnel Split/Merge Application Interface

After we gave the optimize command we received a report that confirmed that the tunnels we intended to split had indeed been split and the ones that we wanted to merge were merged. We checked the router configuration and verified that the changes had taken place.

Figure 10: Figure 10: Cisco's WAN Automation Engine Tunnel Split/Merge Results

Next page: WAN Automation Engine Summary

To go back to the introduction, please click on this link.

WAN Automation Engine Summary
Our tests of Cisco's WAN Automation Engine confirmed that the functions Cisco claimed to support were indeed supported.

We recorded accurate link utilization reporting by the maintenance tool and the bandwidth-on-demand tool. We also confirmed that WAN Automation Engine can configure both Cisco and third-party devices. This is an important aspect for CSPs as nearly all networks are multi-vendor.

As we have seen, WAE enables the engineer to supervise the LSP functions of a network at a higher degree of abstraction.

We also found, both as an independent test lab and subscribers to a network, that the ease with which the bandwidth calendaring and bandwidth-on-demand tools could be operated was very impressive and a great step forward.

With a software tool such as WAN Automation Engine and standardized and defined protocols such as PCEP and REST, the network can be automated and programmed dynamically and on request.

So now, what about the Network Service Orchestrator?

Next page: Cisco Network Service Orchestrator enabled by Tail-f

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator enabled by Tail-f
Another important element in the management and orchestration portfolio Cisco provided to EANTC is the aforementioned Cisco Network Service Orchestrator enabled by Tail-f.

EANTC is very familiar with the Network Service Orchestrator (NSO): While we were validating its capabilities at Cisco's San Jose labs, some of the NSO team members were testing multi-vendor interoperability in our Berlin-based lab. It is rare for EANTC to be testing a product in parallel in two different labs.

So what is NSO? Cisco describes NSO as being akin to a translator software toolbox with two layers.

One layer is a programmable interface to various service tools such as CLI, JSON-RPC, REST, and so on. This layer faces the network operator or other applications: In our earlier processes, WAE was using NSO to program MPLS Label Switched Paths (LSPs) in the network.

The second layer faces southbound, towards the network, and is called Network Elements Drivers (NEDs). This layer uses standards-based interfaces (such as the IETF NETCONF or SNMP) and is augmented by a CLI interface.

However, it is between the two layers where the magic happens, as that is where NSO provides network service programmability. Between the interfaces are two software modules called service manager and device manager. Using these two pieces of software, NSO has the ability to understand service models and what is needed to implement them on a device. Service models are defined in Yang (IETF RFC 6020) which is, as the IETF RFC 6020 explains, "a data modeling language used to model configuration and state data manipulated by the Network Configuration Protocol (NETCONF), NETCONF remote procedure calls, and NETCONF notifications."

This represents a significant change in terms of the way services are created. It changes the focus from defining configuration to defining a service, which NSO then configures on the relevant network devices.

So what can NSO do?

Next page: Cisco Network Service Orchestrator – Service Provisioning

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator – Service Provisioning
We started this part of the validation process with a view of the elements without any services configured. The NSO interface showed five Customer Edge (CE) routers, connected to two provider edge routers from Cisco and two Provider Edge routers from another major vendor.

In the core of the lab network we had an IP router. Our goal was to configure an IP/MPLS Layer 3 Virtual Private Network (L3VPN) service on this multi-vendor, multi-operating system network. (The ability to manage multiple operating systems is a key attribute as even the Cisco provider edge routers were running slightly different operating systems -- one router was running Cisco IOS, while the other had Cisco IOS-XR.)

Figure 11: Figure 11: Network Topology as Discovered by Cisco's NSO

There is not much magic initially in such tests. We had to manually enable access on the routers for the software. We then synchronized the NSO configuration database (CDB) with the device. We noted that Cisco was using two different interfaces to the devices: The Cisco devices used the CLI module, while NSO connected to the third-party router using NETCONF. Why the two approaches?

The Cisco team said they wanted to demonstrate the practical approach of both CLI and NETCONF in a way that a CSP would use them in a live network today, adding that NETCONF could just have easily been used to connect the Cisco gear and CLI used to connect to the third-party router.

We first configured a service called "Volvo" that required a connection from CE1 to PE1 and from CE2 connection to PE2. The activity looked very similar to the configuration of a router, but with a big distinction -- we were not on the router's CLI, but on the NSO command prompt. We then used an NSO command called "commit dry-run" to compare the configuration we were sending to the device with the existing configuration on the device. Once we were certain we were not about to cause any disaster, we applied the configurations.

This is really when the magic happened, because without much effort, with just a small amount of information fed into NSO, it is now possible to make new services come to life without having to type hundreds of lines of code for the router. Instead, NSO has the ability to create services in a higher language, allowing engineers to think about services instead of network element configuration and we were able to quantify that during our validation process.

By adding 15 lines of NSO configuration data, we counted 318 lines of Cisco configuration. We then were able to use the tester attached to the CE routers to send traffic between CE1 and CE2. Since the service model included a 600Mbit/s shaper, we asked the Cisco team to increase the traffic rate to 700 Mbit/s and looked for dropped packets. We indeed lost 100 Mbit/s per direction of traffic, which proved to us that the bandwidth limiter part of the service model was working.

We now had a topology that included two CE routers and two PEs routers, all from Cisco -- so we added two third-party routers to the topology. This time the new service (one CE-to-PE connection) required six lines of configuration in the NSO CLI interface and resulted in 191 lines of device configuration and a functioning service.

The Cisco team explained that the devices were already "discovered" simply by dragging the CE router onto a PE. Again, we sent tester traffic to check that the new device was really added to the service. The results of this initial test were impressive for a few reasons. First, the NSO CLI we were using felt like a Cisco CLI, which was familiar and comfortable. That's because the NSO is able to mimic vendor CLIs: If a network engineer is more comfortable with another vendor's CLI, NSO can adapt to behave like the other CLI, making it easier to use. As a result, multi-vendor networks (once service and device models are applied) could be operated using a single CLI.

The fact that we required so few lines of configuration to create such complex service (not only connectivity but also Quality of Service configuration) also provided a real illustration of the benefits of this software tool. The obvious next question was -- what happens when we want to modify the service?

Next page: Cisco Network Service Orchestrator – Service Modification

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator – Service Modification
Now that we had a fully functioning and tested IP/MPLS L3VPN service running in the test network, it was time to modify some of the configurations in the network.

Using the GUI interface we changed the AS number on one of the CE routers (CE2) and checked that the configuration had taken place in the device itself. Since traffic was actually running between all end points, we could calculate that the service on this particular CE was interrupted for a duration of 6 seconds and then came back to full operation.

We then used the GUI to change a CE connection to a different PE. All we had to do was navigate to the end-point panel, choose a different PE interface to apply to our chosen CE router and apply the change. Again, we could verify the resulting configuration before approving and then commit.

The interesting activity was not simply the change in device configuration -- the whole test bed was sitting on a switch, so changing a connection did not require re-cabling -- but also that NSO cleaned up the configuration that was no longer needed on the initial PE router.

Figure 12: Figure 12: Initial service configuration, all routers participating in the service highlighted in purple by NSO

Figure 13: Figure 13: Resulting IP/MPLS L3VPN service (note CE4 re-homed by NSO)

Next page: Cisco Network Service Orchestrator – Service Restoration

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator – Service Restoration
Now let us assume that the network operations team is still not convinced and that some of the network engineers are actually still modifying router configurations and are not using Cisco's NSO. A mechanism must exist in NSO to identify and hopefully painlessly resolve such issues.

In order to check if NSO could actually deal with such a scenario in a graceful manner, we asked one of the Cisco engineers to log onto one of the CE routers and remove the BGP configuration. Once the engineer typed in "no router BGP," we immediately noted that all test traffic from and to this CE router displayed loss.

When a situation like this occurs in the network, the root cause must be identified and the issue resolved -- that is human nature!

We used the NSO CLI to compare the configuration we had in the network prior to the issue with the CE router. The NSO reported that a section of the configuration was missing. We used a command called "re-deploy" to override the configuration on the CE router. As the Cisco team explained, NSO then compares the service model that should have been on the device with the current configuration and applies only the minimal missing configuration.

We verified that the service was indeed back to normal operations and moved on to the next test scenario.

Figure 14: Figure 14: NSO Command Line Interface in re-deploy state

Next page: Cisco Network Service Orchestrator – Service Restoration Due to Node Failure

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator – Service Restoration Due to Node Failure
The next validation we explored was a situation in which a configuration is ready to be applied but a device that needs to be involved in the service has failed. This time we modified the shaper bandwidth allowance from 600 Mbit/s to 1 Gbit/s. We used the dry-run command again and were provided with a report highlighting the changes to be applied to the configuration. These are depicted in the figure below.

Figure 15: Figure 15: NSO Commit dry-run output with future changes highlighted

Before applying the configuration, we shut down one of the routers involved in the service. Now our commit command was rejected by NSO on the basis that it could not connect one of the devices in the service.

Both service restoration tests showed that Cisco's NSO software is able to react to changes in the network and save an operator from his or her own mistakes. The software also demonstrated mechanisms to allow the operator to easily control the activities that the software intends to perform.

Next page: Cisco Network Service Orchestrator – Device Management

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator – Device Management
Obviously, if a CSP has a large network, adding all configuration commands manually will be a rather tedious and long process and one likely to introduce errors. The Cisco team explained that since NSO understands both services and devices, an operator could create a template for a configuration change and apply it to all devices.

We decided that validating that function would be a rather simple test and asked the Cisco team to activate SNMP on all three device types -- two Cisco IOS devices and a third-party device. Cisco created a configuration that included ten commands in total (the third-party configuration required four lines, while both Cisco IOS configurations required three lines each). Each of the SNMP configuration commands also included a variable, denoted by a dollar sign, with the SNMP community. We then applied the configuration to the device group, providing the community string on the CLI, and watched as in a few seconds, all nine routers were configured with SNMP read-only access and the correct community (public).

This in an interesting and potentially useful concept. We asked Cisco which Element Management System (EMS) was responsible for the devices in our validation network: In fact, NSO incorporates the device management and that since it maintains connectivity with the devices at all times, it can dynamically react to changes in the devices.

Next page: Cisco Network Service Orchestrator – Service Model Modification

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator – Service Model Modification
Last, but certainly not least, we asked for access to the service model used in the tests to see what would happen if we made changes to the configuration.

After all, from the validation processes we had undertaken it was clear that the service model is one of the key elements in the software, which intrigued us. So we modified the QoS model, adding three classes -- bronze, silver and gold -- and reloaded the packages.

Now when we tried to use the NSO CLI to apply QoS configuration to our service, we received the three options that we had just added to the service on all the devices (CE, PE and third-party devices). We verified if the configuration was applied to the devices and were no longer surprised to see that all devices now had the correct QoS configuration modifications.

Next page: Cisco Network Service Orchestrator (NSO) – Tests Summary

To go back to the introduction, please click on this link.

Cisco Network Service Orchestrator (NSO) – Tests Summary
Our NSO-focused tests explored the full service life cycle, from service creation, through modification, redeployment and device management. We were able to measure operational efficiency by showing that with minimal NSO configuration commands, we could create extensive device configuration.

Of course, we cannot attest to the efforts involved in the creation of the service and device models: That might be a good topic for a follow-up validation process.

We also confirmed Cisco's statement that NSO is able to provision multiple devices, not all of them from Cisco. Configuring multi-vendor, multi-generation operating systems is a novel way to deal with brownfield deployment as well as being very much in line with the drive towards SDN. In fact, the creation of services through templates looked a lot more like programming the network than configuring the network.

Next page: Step 3: The Cloud Services Creation and Delivery Stage, Cisco Cloud Managed VPN Solution for Virtual Managed Services

To go back to the introduction, please click on this link.

Step 3: The Cloud Services Creation and Delivery Stage
Both Cisco's WAN Automation Engine and Network Service Orchestrator are software tools designed to enable a CSP to automate and increase the speed of operating network devices as well as rolling out new services.

As Cisco explained, both tools can be used to completely automate the deployment of new services. The logic behind the next applications-focused validation process was that, according to Cisco, CSPs have not been able to address the Small and Medium Business (SMB) market effectively, since the cost of customer acquisition and rollout in that market segment is prohibitive.

With greater control and flexibility, though, the market dynamics can change, especially through the use of self-provisioning tools. Market dynamics are not something that EANTC, as an independent test lab, can comment on. What we can do, though, is test the ability of an end-customer to provision a service and to use it without having to interact at all with a CSP call center and without burdening a service provider's operations department. To achieve that, the whole service life-cycle process -- from ordering through service delivery, service management and quality assurance -- must be completely automated.

Cisco Cloud Managed VPN Solution for Virtual Managed Services
Cisco presented us with a system it created for one of its Tier 1 CSP customers. The service was completely branded with said CSP's logos and other corporate identity, which is why the figures we display in this section are actually taken from dCloud: The same service exists as an evaluation scenario and was part of the dCloud verification.

The scenario we were trying to build is a small business of two locations that wants a VPN. We called the business (fondly) "Lakshmi's Café." The perspective of this process shifted from CSP to end user, the owner of the café. Acting as the owner, we were interested in connecting both locations using a VPN as well as making sure that the free WiFi we intend to provide in our imaginary café was protected.

The whole process of ordering the service was driven by a web interface. We started by creating a new customer account for our café. As you can see in the figure below, there were several options on the menu. We choose the Cloud VPN Full service since it included Firewall, URL filter, Remote Access, and Cloud VPN with Internet Access.

Figure 16: Figure 16: Cisco Cloud VPN Service Type Menu

Once we clicked to the next page we had to insert the number of sites we wanted to connect and the number of users at each site. We could also indicate the expected staff growth, which the Cisco team explained was used for bandwidth calculation.

Figure 17: Figure 17: Cisco Cloud VPN Company Profile Setting

After this step was complete, we understood the point of some of the questions. The price for the basic service was displayed on the first page. Based on the company profile, additional bandwidth could be added, at cost, to the service. Cisco uses the company profile data to recommend what they think will be appropriate for the customer size. We decided to stick with the recommendations (remember, we are a café owner here, not network engineers) and moved on to the next page.

At this point Cisco explained that the speed is actually part of the license. This meant that the virtual router that will be associated with the service will have a specific license associated with it.

Figure 18: Figure 18: Cisco Cloud VPN Speed Selection Panel

The next panel allowed us to choose the type of SSL license. This is where could imagine that small and medium businesses would run into difficulties. What is SSL VPN and why would I need it? That would be the first question that would come to mind. In this case, since we did not know what SSL VPN was, we agreed with the most basic recommendation, using the logic, "If you don't know what it is, you don't want to pay for it."

Cisco explained that the SSL VPN connection was actually part of the secure connection to the cloud and that it was required to secure the customer access to the service.

Figure 19: Figure 19: Cisco Cloud VPN SSL License Panel

This is really where the service began to take shape and the power of virtualization came into play. The next panel allowed us to choose URL filtering options. As a café owner, we liked the idea that the folks drinking at our establishment will not be able to visit malicious or illegal websites, nor did we feel that our coffee shop should allow access to adult material. We picked the medium-level URL filtering service and turned our attention to Cisco.

Our question to the vendor was simple: How will this work? Cisco explained that once the service is ordered, its Web Security Virtual Appliance (WSAv) will be instantiated in the CSP cloud and will be responsible for performing the URL filtering function. With that information in mind, we continued with the ordering process, while we also informed Cisco that we'd like to record the whole process happening in the background once our order was completed.

Figure 20: Figure 20: Cisco Cloud VPN URL Filtering Level Panel

And a moment later, we were done! Once we moved on to the next panel we received a list of two devices that were going to cost us a lot of money. Since we had two sites and the devices also included WiFi Access Point function, we felt that our café could benefit from these elements and we moved on to the next panel. The latter was an Amazon-like shipping address system that allowed us to send a router to each of our cafés. We then received a nice "thank you for your purchase" panel and, with this, the Cloud VPN service for Lakshmi's Café was purchased.

Figure 21: Figure 21: Cisco Cloud VPN Device Panel

Figure 22: Figure 22: Cisco Cloud VPN URL Filtering Shipping Address Panel

Next page: Cisco Cloud VPN: Behind the scenes

To go back to the introduction, please click on this link.

Cisco Cloud VPN: Behind the scenes
Once the service was added, the next step was to wait for the routers to be delivered. We skipped this phase, of course, and proceeded to the part where the service actually spins up.

As Cisco clarified, once Lakshmi's Café receives the small routers, the serial number of the routers will be manually inserted into the same web interface panel. This will start the process of services instantiation. We inserted the serial number to the web interface and switched the view to the command line interface to monitor what was happening.

The log scrolled for roughly 20 minutes until the system announced that the service was ready to go. This included spinning up a Cisco CSR1000v, a Web Security Virtual Appliance (WSAv) as well as a Cisco ASA 1000V Cloud Firewall. As Cisco revealed, all virtual functions were instantiated over Kernel Based Virtual Machines (KVM).

We then used Cisco's Network Services Orchestrator (NSO) to take stock of the devices, virtual and otherwise, to verify that everything we ordered was not only instantiated, but also manageable. Last, but not least, we verified there were two IPsec tunnels established using the CLI of the two routers. Cisco explained that one IPsec tunnel was used for management (connected to NSO) and the second tunnel was used for user data. The logic behind this is that Cisco assumes that everything is over-the-top, which means NSO and other management access will have to take place, possibly, over the Internet (hence the management IPSec tunnel).

Last, but not least, we took two more actions: We added the second café to the service and at the end, deleted one of the sites from the service. We performed both activities from the web interface and used Cisco's NSO to verify both.

Next page: Cisco Cloud VPN Solution Life Cycle Conclusion

To go back to the introduction, please click on this link.

Cisco Cloud VPN Solution Life Cycle Conclusion
As Cisco explained, the whole backend system, which allows the service to be completely automated, is an instantiation of ETSI's NFV ISG architecture. If we try to map the elements we saw in this test, we see that the following elements map nicely to the NFV architecture:

From an end customer perspective, in this case the owner of Lakshmi's Café, we had a great experience. Without calling any support or ordering line, using a web interface only, we created a service that included not only connectivity, but also services. Remaining with the same perspective, we measured 20 minutes to get the whole service up and running, including all added functions (such as a firewall).

From a service provider perspective, a customer created a rather complex service without involving an operations team. Using the cloud, the various automation tools we already tested and open source software (OpenStack), no humans, other than the shipping department, were involved with the service rollout.

Another observation is that rolling out additional services, with more sites, would actually require the same amount of time, since the instantiation of virtualized services should remain a constant -- they only differ from each other by licensing initially. This will be an excellent point to validate in a follow-up test.

Next page: Production Environment Application Demonstrations – Project Squared and Mobility IQ

To go back to the introduction, please click on this link.

Production Environment Application Demonstrations – Project Squared and Mobility IQ
The final piece of the puzzle was a hands-on demonstration -- but not a test or validation -- of two software-as-a-service (SaaS) applications that Cisco has developed for its customers. As Cisco explained, both software tools were running in a live production environment, which is why we could not get "hands-on" with them.

Cisco's Reinhardt Quelle, operations architect of the company's Intercloud (its "cloud of clouds"), explained that Cisco has been committed to creating an open cloud environment for a while. He even backed up his statements with a blog entry on the OpenStack web server that explains how Cisco is using OpenStack. As Reinhardt explained, the real magic, and the value proposition such applications have for a service provider, is that they actually "just run" -- service providers do not have to spend capex on components to create the service.

As Reinhardt was walking us through the whole production system, which included Cisco's own data centers as well as public cloud access and CSP clouds, we noted that a node just came up in one of the Intercloud's service providers. It is no secret that Cisco signed several service providers around the world to be part of the Intercloud, as Light Reading reported several times during 2014. Nonetheless, seeing a node come up, during a discussion on the system, was a nice coincidence.

As reported in September 2014, Deutsche Telekom (DT) has also been part of the Intercloud ecosystem. Since EANTC is also a German company (though clearly not the same size as Deutsche Telekom), we were interested in data governance. After all, in Germany, privacy laws are very different than in the US. For example, a service provider such as DT has to be able to maintain control over the data used in any services and guarantee that the data remains in Germany. Reinhardt explained that data governance as well as encryption and security are big aspects of Intercloud. We wanted to try and move data from Germany to other countries, but Cisco reminded us that the system is live and recommended that such tests should be done at a later date.

So instead of talking about theoretical aspects of Intercloud we asked to move to the applications running on top of Intercloud. Cisco reminded us that we already had our hands on one Intercloud-supported application -- dCloud. They added that the applications we were about to see were created by Cisco as a way for CSPs to further sell SaaS apps to their customers.

Cisco WebEx Squared
Synergy Research places Cisco's WebEx as the global leader for SaaS web conferencing, with 51% market share. As collaboration tools evolved over the years (adding functionality such as file sharing, search and encryption), so did the resources they consume. For most networked enterprises, collaboration tools are also mission-critical, which puts the emphasis on resiliency and availability. All these were reasons for Cisco to work on an evolution of WebEx. The new collaboration tool is currently called Project Squared and is already available on the web as well as in the iTunes Store and Google Play.

We installed Squared on our smartphones and also moved the whole conversation to the web interface. Reinhardt explained that because Squared is running over multiple clouds, when the recent Xen hypervisor bug hit (forcing RackSpace to reboot), the Squared service was not affected at all.

We can confirm that Project Squared is real and so can you, by simply downloading the applications and using them. We can also confirm that Cisco, based on the web-based management system they showed us, has an extensive infrastructure spanning a large number of clouds -- public, CSP-based and their own. We certainly hope that in the next phase of testing we could spend more time and delve into this infrastructure.

Mobility IQ
Cisco has just released Mobility IQ. This SaaS application is designed to offer deep visibility into small cell and WiFi access. The app is targeted at CSP customers that, in turn, are supposed to offer this service to their customers.

And what does the service bring? Well, Cisco was careful to pick a customer it said was live and an operator of a major sports and concert hall in a large metropolitan area. The tool provided statistics about technical key performance indicators (KPI) such as WiFi attachment rate, access point health, bandwidth usage, spectrum usage, and so on. Alongside these network-centric KPIs, Mobility IQ offers answers to consumer questions such as "Which floor is busier?" or "Which bathroom is more packed?" The theory with the latter metrics is that an operator of a service within the WiFi area could attract customers by sending discount notifications to the end-users.

Figure 23: Figure 23: Mobility IQ Zone Movement Example

Cisco demonstrated an extensive and slick-looking web interface that provided an overview of the various KPIs and allowed the CSP to further tailor the view and provide access to an organization level. The interesting aspect for us, other than the details of how this works, was the claim that Cisco made about the way it hopes the service will be used by CSPs: Cisco intends to sell Mobility IQ to service providers and perform the whole integration, while service providers will "only" need to re-brand the service.

Again, since the system was live, we could not gain access to such elements as the statistics collection or actually simulate WiFi access point failure. We can confirm that the system appears to be real and operational and we hope, as with Project Squared, to see the system in action during a future test.

Next page: Conclusion: Meaningful Developments

To go back to the introduction, please click on this link.

Conclusion: Meaningful Developments
EANTC has been a contributing member of the ETSI Network Functions Virtualization Industry Specification Group (NFV ISG) since its third meeting in July 2013. The ISG has released three white papers between its formation outlining the vision, drives, progress and results. As the concepts of NFV started taking flight, the ISG's CSP members communicated that they were "concerned to maintain momentum by avoiding a protracted standards effort."

In this report we were able to verify that Cisco's virtualization solution is real. As we showed in the report, Cisco's Cloud VPN appears to adhere to the NFV architecture. We also verified that the Cisco solution works with at least one third-party device and uses free software and standardized interfaces, as defined by the IETF, to orchestrate and manage services.

This report shows real and tangible momentum, based on existing standards and code: It just happens to come from the IP sector's giant. Chapeau.

— Jambi I. Ganbar, Senior Technical Marketing Manager and Carsten Rossenhövel, Managing Director, European Advanced Networking Test Center AG (EANTC) (http://www.eantc.de/), an independent test lab in Berlin. EANTC offers vendor-neutral network test facilities for manufacturers, service providers, and enterprises.

To go back to the introduction, please click on this link.