To find the answer, we started our investigation where we left off in our 2007 Cisco IPTV independent test report – Testing Cisco's IPTV Infrastructure – and expanded it to create three types of networks: video contribution, secondary distribution, and a converged network serving residential and enterprise customers.
Cisco brought to the test its arsenal of routers: the CRS-1 core router (which we've also tested before – see 40-Gig Router Test Results), the recently announced ASR9010 for aggregation, and the 7600 for the DSL- and cable network-facing service edge.
In the 2007 edition of the test, we had already base-lined the standard routing, quality-of-service, failover, and multicast tests, so we avoided repeating them. It would have been nice to baseline everything again just to make sure the latest hardware and software does not fall behind previous versions, but, honestly speaking, we did not make friends with Cisco by being pedantic like an incumbent operator. So one could argue that the baseline tests are missing for newer Cisco equipment – the ASR 9010, specifically, as the world has not seen full-feature support in any independent public test yet. Let’s say we leave these tests to the reader as an exercise.
Looking to the future
While in previous Light Reading tests we insisted on using only production-grade code, we allowed Cisco to put beta software and "first look" hardware to the test in some cases this time. All test results achieved with such configurations are clearly marked throughout the report. The rationale was that having a glimpse into the near future seemed worthwhile. We know that Tier 1 operators have already started testing these products earlier this year. Future-proofing the IP video network and services provides a competitive advantage:
- In most markets worldwide, consumers and enterprises are only a phone call away from switching to another service provider. The revenue per triple-play consumer or enterprise video customer is higher than for the average broadband Internet-only customer, so it is important to win such customers with superior technology.
- Video subscribers are accustomed to a certain level of service as traditionally provided by the cable or satellite TV providers. It is ironic that some operators have to guarantee more stringent QoS and resilience to triple-play consumers nowadays than they were ready to offer to enterprise customers a while back.
We selected the tests of Cisco’s IP video transport solution for this report to validate the solution’s flexibility, scalability, in-line monitoring, and resiliency when dealing with IP video applications.
EANTC test requirements
Throughout our testing, we enforced one major EANTC requirement: realism. Before accepting a test, it had to be well understood what its applicability would be to a real service provider network designed according to the vendor’s best practices. Could a reader answer the questions: "Why is this test important for my business case?" and "How can I use the results?" Once this was understood, we got to work designing the test traffic to be used in order to create authentic network characteristics.
This test was Cisco’s most complex and realistic public testing effort in 2009 and probably ever. Formal testing took only two weeks in April and May, but it was preceded by an extensive Cisco preparation phase that started in November 2008. A large amount of effort was given to coordinating all the product groups and business units involved in the test. More than eight of Cisco’s carrier-class routing and switching products were involved in the service delivery network testing – some of them brand new and unannounced at the time of preparation. It took Cisco a little while to firm up the detailed network architecture fitting all the different components. The result was worth it, and we were able to match the network design requirements a modern, cutting-edge service provider would expect to see.
Third-party testing is bound to face scrutiny and be subject to the questions of critical readers, and all for good reason. It is with this in mind that EANTC defined a set of testing rules before the project proceeded. First, we agreed that EANTC control the test plan. Cisco professionally agreed to follow the independent test guidelines, and the final decision was always held by EANTC.
We also divided the responsibilities strictly between EANTC and testing vendor Spirent Communications plc , on one hand, and Cisco on the other. EANTC and Spirent were to operate all testing hardware and auxiliary devices (such as wire-monitoring, or impairment devices) while Cisco was to configure the complete network under test, including any Cisco components, and, if needed, troubleshoot any problems found in the network while testing.
Once testing started, EANTC changed all passwords in the network and maintained control of the devices. We logged CLI output and captured the configuration files for each of the tests and every element within the test. Given the complexity and the number of tests executed, we ended up with a good amount of Cisco configuration files for Cisco’s three primary operating systems: IOS, IOS XR, and NX-OS.
Next Page: The Service Delivery Network