That would be a dirty little secret, if the secret were little. Or even a secret.
A critical baseline enabler for any technology is test & measurement, in conjunction with T&M's kissing cousins monitoring and assurance. But the T&M/assurance industry is stuck in a chicken/egg paradox, orchestration-wise.
The vendors of orchestration software still aren't in concert (not sorry about that at all) on what orchestration software even is, so how can test vendors build commercial tools or standardized test routines to test… whatever orchestration is?
Enter standards organizations such as ETSI, for example, which is trying to shepherd the development of an open source NFV management and orchestration ( MANO) software stack. ETSI's ambition with this project is to "increase the likelihood of interoperability among NFV implementations," according to the operation's website.
Increase the likelihood of interoperability…
How encouraging is that?
So far it's been up to people who are involved with early interops and plugfests to develop test tools. EANTC is a test organization that has worked with ETSI to arrange some recent MANO interops (ETSI is prepping a summary on the results to be released soon -- watch this space).
EANTC Managing Director Carsten Rossenhoevel said his organization helped ETSI create a tool that can adapt flexibly to test the interfaces of MANOs and the underlying NFV infrastructure in an automated way.
OPNFV is working separately on similar goals . OpenDaylight too. AT&T Inc. (NYSE: T), as the lead developer on ECOMP (which it has made open source). But none of these organizations are talking to the others, Rossenhoevel noted. (See Has AT&T ICE'ed VNF Onboarding?)
Complicating the situation, there doesn't seem to be any getting around having to test not only every combination of MANO with NFVi, but also every combination of MANO with NFVi and every NFV. It's a laborious process, hence the desire for automating it all.
"All look at their own code, but nobody's looking at the MANO side for this level of testing yet," Rossenhoevel reports. "Everyone in our recent tests was doing all of this manually with graphical user interfaces -- it's not automated."
And so far they can only test offline. There's no way to test operating networks.
Post-deployment is the killer. It's one thing to set up a plugfest and jerry-rig some test tools. Post-deployment, all bets are off.
I spoke with Ari Banerjee, vice president of strategy for Netcracker recently. Post-deployment, he said, testing companies have no role at all today. They simply lack the experience, he explained. It's up to companies that manage the deployments to figure it out.
Netcracker's provenance is in operations support systems (OSS), and that's the type of expertise currently required, Banerjee said.
Indeed, the growing synergy between monitoring and assurance on the one hand and OSS on the other is a well-remarked phenomenon, underlined by events such as AT&T choosing OSS/BSS specialist Amdocs Ltd. (NYSE: DOX) to be the system integrator for the ECOMP system, which AT&T helped define but made open source.
So for test companies, the issue is that there's as yet nothing standardized to test -- the market, as such, borders on chaos. We'll be talking with the test and assurance companies in coming weeks to get their views on how the market might sort itself out, and how it might be possible to provide automated commercial tools for testing orchestrators.
— Brian Santo, Senior Editor, Components, T&M, Light Reading
CALLING ALL TEST, ASSURANCE AND MONITORING COMPANIES: Make sure your company and services are listed free of charge at Testapedia, the comprehensive set of searchable databases covering the companies, products, industry organizations and people that are directly involved in defining and shaping the telecom test and measurement industry.