& cplSiteName &

Time for a Telecom Reboot

Steve Saunders
4/19/2017
92%
8%

May you live in interesting times.

That apocryphal Chinese curse provides a fitting label for the 2017 telecom market, as we find ourselves mired in an unprecedented mess -- one that's bad for business for everyone in the entire supply chain of next-gen communications.

The root cause? Virtualization! (In both its NFV and SDN flavors.) For years, the industry has been "bigging up" virtualization as the next great telecom revolution, a way to build cost-effective software-based networks to support the emerging digital global economy. But so far the only CSPs that have been brave -- or foolish -- enough to deploy virtualization have found it to be an expensive and time-consuming exercise in frustration, one that has yet to deliver on either cost savings and or new revenues.

To fix this situation we first need to understand what's gone wrong: For 150 years, the telecom industry has owed its success to a clearly defined and proven process for standardizing new network technologies. Then along came virtualization and for some reason we got it into our heads that it would be a great idea to just up and abandon our standards-based best practices altogether, instead developing virtualization by punting it over the fence into a baffling array of open source groups and industry consortia (each loaded with a full boat of personal, corporate, and political agendas, incidentally).

This was a particularly inauspicious start -- and here we are, after four years of intensive work on NFV, and we're really no closer to figuring this thing out than when we started -- with no strong industry standard defining it. (ETSI's efforts don't count; being akin to the "Code" in Pirates of the Caribbean… more of a guideline really.)

Why is this a problem? Because without standards there can be no certification testing; without certification testing there can be no interoperability; and without interoperability service providers are in the same place they were in the 1990s: locked in to buying overpriced proprietary solutions from incumbent equipment vendors.

Further, without the competitive kick in the pants created by a competitive, heterogeneous, interoperable environment, vendors have failed to deliver solutions that are fit for service providers' purpose. Today, it takes an average of six months for Tier 1 service providers just to get the NFVi code from the industry's leading infrastructure vendors to work -- not really what anyone would call "out of the box" software. And typically service providers have to pay the systems integration division of the NFVi vendor to do that work… talk about jobs for the boys.

So that's half a year of expensive integration work just to get the NFV plumbing to work.

But wait -- there's worse! The value of virtualized networks is supposed to lie in the magnificent variety of new services that run over them. But the fact is that NFVi vendors have been paying lip service to a vague concept of "openness" while simultaneously maintaining the proprietary software and hardware hooks that have kept them profitably in business for the last two decades. Which means that at the end of that six-month installation period, carriers are finding that the only services they can run over that infrastructure are the VNFs that are sold by -- correct! -- the same company that sold them the infrastructure.

This is:

1. Not good enough!

2. The exact opposite of what service providers were told to expect from NFV.

Making things worse, issues within the virtualization sector are having a corrosive effect on the commercial prospects for the wider telecom industry. Growing frustration on the part of CSPs with the shoddy state of the technology has prompted them to push their virtualization plans back -- or postpone them altogether. That's left mega-corporations like Nokia, Cisco, and Ericsson with a big fat hole in their bookings ledgers where those sales of virtualization technology and services were supposed to sit. And that, in turn, has sent an icy chill through the rest of the telecom ecosystem, derailing the growth and sending the industry into what is starting to feel more like a death spiral than a market correction.

So who's to blame for this situation?

Let's start with the open source community. With its insanely complicated, quasi-anarchic, happy-clappy, "we don't like standards" approach, open source's approach to developing code works fine if you're crowd-coding an Internet browser (or a spot of freeware Donkey Kong) but the effect of the open source process on the telecom market has been toxic. What we need to do is take the innovation and the ideas from open source and then superimpose telecom's unique brand of discipline and standards over them -- something which simply has not happened yet.

Let's take a moment to wag the giant foam finger of admonishment at enterprise vendors, also. They leapt -- like a salmon! -- on virtualization as an opportunity to compete with their telco vendor counterparts to start building carrier networks, arguing that what we're really talking about here is building virtualized cloud networks. "Nobody builds cloud networks like we do," they Trumped. The problem with that line is that enterprise cloud and telco cloud have totally different requirements, and it turns out that enterprise vendors are actually a bit shit at building telco nets (see Telefónica Ditches HPE as Virtualization Lead). (This should not, perhaps, come as a huge surprise. HPE has been a bit shit at doing anything other than being a bit shit for as long as most of us can remember. The idea that it could suddenly and overnight -- hey presto! -- become not shit at building the largest and most demanding telecom networks in the world was always questionable.)

Trade publishers (ahem), analyst firms (sorry) and so-called experts in general also should be called out for hyping the possibilities without paying more attention to the realities.

But the demographic that must take most of the blame for the current very real virtualization cataclysm is, of course, the telecom community as a whole -- for allowing ourselves to be wooed by the promise of virtualization and abandoning the first principles that have successfully guided us, as an industry, since the 19th century. How do we get back on track from here? As an industry, we need to stop the crazy train and get back to basics.

That process starts with defining realistic goals. I've heard a lot of hoo-hah over the last few years about how the end point for service providers is a DevOps environment like the one found in enterprise networks. This is, to use a technical term, complete bollocks! Excepting OTTs, the vast majority of service providers and CSPs have neither the culture nor the skill set to implement DevOps -- even if they wanted to. And they don't. One of the supposed benefits of a DevOps environment is that it allows constant changes to be made to network services. That's fine in an enterprise world, if you like that kind of thing (and enterprise IT departments seem to live for it) but on a telecom network it's just about the last thing CSPs want to deal with.

What service providers actually want is what they were promised when NFV first emerged: specifically, to be able to implement the best and most popular (and most profitable) mix of services by picking and choosing from an online marketplace chock full of "best in class" services and applications, developed by specialist third parties, in the sure and certain knowledge that these VNFs are absolutely guaranteed to run over the NFV infrastructure they have in place. Creating this virtualization free market technology economy is not as hard as we've made it look. But it will require us, as an industry, to pick one API (one, yes? As in, less than two, FFS!) between the NFVi layer and the VNFs that run over it, and standardize on it.

FYI, for the last six months, working alongside the not for profit New IP Agency (NIA), I've been reaching out behind scenes of the telecom industry to gather the support required to standardize on just such an API specification, and launch an independent certification program based on it.

I'll be sharing more information about the NIA's plan in my column here in a couple of weeks' time but I can promise you that it will be very good news for the industry -- probably the best news we've had in this very challenging year, inshallah.

— Stephen Saunders, Founder and CEO, Light Reading

(45)  | 
Comment  | 
Print  | 
Threaded  |  Newest First  |  Oldest First        ADD A COMMENT
mendyk
100%
0%
mendyk,
User Rank: Light Sabre
4/19/2017 | 10:09:48 AM
Woeful
The telecom industry is at the heart of the 21st-century global economy. There's no Google, no Amazon, no nothin' without CSPs enabling all this digital transformation stuff. And yet right now this sector gives off a distinct Cleveland Browns vibe. Woe is us. We can't win. We're stuck in a two-bit role. Nobody likes us, and nobody is helping us. Enough already. Transformation is hard. It requires work, and commitment, and patience, and money. Those are all factors that CSPs can do something about. Whining accomplishes nothing positive.
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/19/2017 | 4:25:38 PM
Re: Woeful
I hear more whining from Open Source people than CSPs. 

The richter scale of behaviors is as follows: 

Enterprise vendor: arrogant 

Telecom vendor: worried 

CSP: angry and worried 

OTT: entitled and oblivious 

Open Source: high and oblivious 

 

 

 
brooks7
50%
50%
brooks7,
User Rank: Light Sabre
4/19/2017 | 10:50:58 AM
Diagnosis
 

I think there is exactly one problem.  You can not take a technology and in fact an entire infrasructure that was built to be used in one way and then decide that the entire industry would change to do it another. 

The virtualization deployment that we see in the IT world grew up organically and all the stuff that you talk about that is hard or expensive is already addressed.  The problem is that this model was completely rejected.  I have run a SaaS operation and it works nothing like a carrier.  I think people need to think Scrum and Agile.  10 groups have products out before the Telcos have an approved business plan.

This is not a world where standardization and interoperability matter in the least.  Neither does bugs in Open Source.  You start with those things and then your software team patches the Open Source and integrates things into a product.  The fact there is a standards body or 2 or 3 means that this whole effort is a failure. 

Then there is a second problem.  What is the gain for the telco vendors to make their products even lower price?  To take full advantage of a Virtual Environment, vendors would have to redo their products from scratch.  How is that supposed to work economically?

So if I back up to what worked so well on the IT front, essentially a couple of services are built this way out of a common set of core blocks that are used by lots of folks (LAMP Stack + VMware).  On top of the web scale providers do their own thing to make it work in their specific situation.  Each of these services are run and implemented in a way that is very different than what telcos are used to.  I have personally updated the software that operated the service we ran more than 1 time in a day (we had about 3.5M end users).  Imagine that in Verizon or AT&T.

seven

 

 
Carol Wilson
100%
0%
Carol Wilson,
User Rank: Blogger
4/19/2017 | 11:39:30 AM
Re: Diagnosis
Brooks7,

How do you view the way Facebook and Google are using virtualization - I see CSPs much more interested in emulating their processes, which do include daily updates. 

I totally agree on the vendor issue - what I am hearing from operators is that the vendors are resisting the kinds of pricing restructuring that is needed because it forces them to completely redo their business plans. 

It was mentioned many times at ONS a few weeks back - the folks being expected to enact the change have no incentive to do so. 
brooks7
33%
67%
brooks7,
User Rank: Light Sabre
4/19/2017 | 1:41:45 PM
Re: Diagnosis
 

Carol,

I think you misunderstood my daily updates.  Ask Verizon if they could introduce new packages from their vendors in 24 hours and put them live to their market.  The time to test and approve things is months not hours.  Very different timeframes.  The Scrum methodology produces 2 week product release cycles for new features.  Facebook and Google basically did what I said...picked their vendors (sometimes it was them) and just DID.  They didn't wait for standards development.  They did try to get what they were doing standardized, but only at the very bottom end of things (like component specs).

By the way, I don't think that this has anything to do with what the telcos want.  What they want is to reduce capex.  They want open source because they would like to eliminate the spending on equipment.  That is NOT why the IT groups do virtualization.  They do it for fast service creation and flexibility.  Facebook's (and Google and Amazon) have the load on their network change daily with the movement around the Earth by the Sun.  It is either be flexible or double the amount of equipment in each data center.  THAT is their CAPEX reduction.   The say AT&T equivalent is that they have 4 voice switches in the US and they use the West Coast Data Center to cover overflow on the East Coast Data Center every day from 8 - 11AM.   Now scale that up and voila...now you are starting to talk.  Of course, none of this works if you have to have special network arrangements or any oddball protocols.  You want to use DNS publishing and load balancers to make this work.  Imagine if your local phone call was actually handled in the early morning by a switch that was based in California.  That is pretty much like making the equivalent of loading up google.com and thinking you know where the server you are actually connected to for that TCP connection is located.

seven

 
Carol Wilson
100%
0%
Carol Wilson,
User Rank: Blogger
4/19/2017 | 2:01:00 PM
Re: Diagnosis
I get what you are saying but the work that AT&T, Vodafone and others are doing is pushing in the direction of Facebook and Google. 

I disagree that telcos want open source primarily to reduce capex and what they are spending on equipment - that's a goal, to be sure, but what I hear over and over from them is the need to scale more rapidly to meet bandwidth demands and to introduce new services faster. Not a single major CTO with whom I've spoken thinks virtualization will reduce their capex budgets any time soon. 

 
brooks7
50%
50%
brooks7,
User Rank: Light Sabre
4/19/2017 | 2:13:49 PM
Re: Diagnosis
Carol,

I think that is the mistake...forget scale of services.  How about introducing 10 new services a day.  And I disagree that they are pushing in the direction.  They would stop having standards bodies and start hiring software engineers TODAY. 

seven

 
Carol Wilson
50%
50%
Carol Wilson,
User Rank: Blogger
4/19/2017 | 3:17:47 PM
Re: Diagnosis
So when they tell me they are hiring software engineers, that the issue is finding them, that's not true?

As for standards bodies, there is much greater engagement with open source groups today - every major operator I know is sending developers to groups like OpenDaylight, ONAP, TIP and more. 

I get the skepticism, it's never much of a risk to say these companies are behemoths that can't change. But if these guys aren't pushing in this direction, why are they spending so much time and energy claiming they are?

 
brooks7
0%
100%
brooks7,
User Rank: Light Sabre
4/19/2017 | 6:23:18 PM
Re: Diagnosis
It turns out there are hundreds of startups in SF.  So...how can they have lots of SW folks but the SPs can't find them?  Facebook, Apple, Amazon, Google, etc...ALL have software teams.  Heck, they could buy Wipro or one of the other Outsourcers from India if they wanted engineers.  I would view that whole notion with complete skepticism.   They have been at this for YEARS.  And they can't hire...did the dog eat their homework?

I would want to ask them the following:  Can you share with me the open requisition numbers for these jobs?  They are posted on the website right?   I was poking around AT&T and for this kind of initiative I would expect to see 10,000 or so jobs open.  If they won't share the job reqs...how about what section of the career site that it is.

seven

 
brooks7
50%
50%
brooks7,
User Rank: Light Sabre
4/19/2017 | 7:13:52 PM
Re: Diagnosis
Steve,

I actually disagree.  All the major CSPs are making money.  As far as I can tell, they are mostly concerned that somebody else (read Google) is making more money than they are and they are mad about it.  But as far as I can tell in the US that Verizon, AT&T, Comcast, Charter, Centurylink, Cox and Frontier are all quite profitable.  The little carriers will have problems with USF exits stage left and many CLECs are problematic (like Integra).  But there are plenty of small ISPs, WISPs and small carriers in general that are doing quite well.

So here is my list of things that need to change: (Null Set).

Vendors on the other hand are in a depression and have been since the collapse of the CLEC bubble in 2000.  Just think about how much infrastructure between Wireless and Fiber is being installed.  Now tell me why Bell Labs belongs to Nokia?  The equipment vendors have failed to adapt in the changes post the bubble.  They are led around by their "equipment" by the Service Providers and develop TONS of things that go nowhere.  If you go into an SP, you will find dozens of people who are happy to talk about all the things that they are going to do. 

Now, should Service Providers form a division that does some applications and such.  Well, maybe.  But they can not evaluate the same way they do Network Investments.  Other than that, they will start change when they are losing money.  Until then, it will be like installing FTTH in Europe.  I was on panels in 2003 where people were telling me that this was the year.  Its 2017, and maybe next year will be the year.

seven

 
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/20/2017 | 12:04:31 PM
Re: Diagnosis
"I actually disagree.  All the major CSPs are making money.  As far as I can tell, they are mostly concerned that somebody else (read Google) is making more money than they are and they are mad about it.  But as far as I can tell in the US that Verizon, AT&T, Comcast, Charter, Centurylink, Cox and Frontier are all quite profitable.  The little carriers will have problems with USF exits stage left and many CLECs are problematic (like Integra).  But there are plenty of small ISPs, WISPs and small carriers in general that are doing quite well."

 

Interesting! 

 

I shall have to ask the LR editors about the profitability or otherwise of the big CSPs!
litesabre
100%
0%
litesabre,
User Rank: Light Beer
4/19/2017 | 7:16:17 PM
Re: Diagnosis
seven: "It turns out there are hundreds of startups in SF.  So...how can they have lots of SW folks but the SPs can't find them?  Facebook, Apple, Amazon, Google, etc...ALL have software teams."

But the problem is that an internal development organization is a difficult beast to manage.  You have to be built for that, like Google or Facebook, i.e., to eat your own cooking.  You have no leverage, the historical advantage of the SP over the vendor (how many times have I heard the "but vendor X says..." or "I guess I'll have to go the RFP route..." if I object to some dumb SP request.

That's why an OSS system built in-house takes an act of God to modify to release a new feature.

And the insane belief at SPs that all software should be free, and only hardware costs money.

SPs don't know diddly squat about how to build software.  They wouldn't know what to do with the software engineers if they poked them in the collective SP eye.

-litesabre
brooks7
100%
0%
brooks7,
User Rank: Light Sabre
4/20/2017 | 2:10:31 AM
Re: Diagnosis
lightsabre,

My point is that the entire IT world doesn't just expect its OS vendors and other players to conform to a bunch of interoperable standards at the level that the Carriers are trying to do with NFV.  What they have done is hire people and make the software that they have work for them.  They patch over issues with software and other off the shelf packages. 

My view has been for a long time is that the right way for the SP providers to move forward is to dump almost all the folks that work on this kind of stuff and simplify their networks.  Eliminate the barriers inside and try to reduce the technologies involved by about 80%.  If that means you don't do stuff, well then you don't do it.  That kind of streamlining will add much more to the bottom line than anything else that they can do.  They can't turn the network into something other than a commodity.  That ship has sailed and they (and the vendor community) have wasted Billions of dollars trying to change that.

seven

 
Carol Wilson
50%
50%
Carol Wilson,
User Rank: Blogger
4/20/2017 | 3:07:01 PM
Re: Diagnosis
Seven,

There was a variety of companies at the Open Networking Summit - smaller ones including startups along with Google, Facebook, AT&T, etc. - the one thing on which they all agreed is that there is much more demand for software engineers than exist today on the planet. Are a lot of the smarter ones more interested in starting their own companies than working in the bowels of a telecom giant? I'm sure that's true.

For telecom players to try to scale their networks in the way that Facebook and Google do isn't ludicrous and it isn't about them trying to be an OTT player. It is about trying to meet bandwidth demand in a way that scales. And the traditional approaches don't do that. 

But this discussion seems to be much more about bashing telecom in the most outrageous way for other purposes. 
brooks7
100%
0%
brooks7,
User Rank: Light Sabre
4/20/2017 | 4:57:32 PM
Re: Diagnosis
Carol,

I think they might be interested in scaling their networks.  So,  Google and Facebook have done it.  There is existence proof that NOTHING is stopping anybody from doing it.  So stop waiting and do it.  There are no barriers.  The telcos are inventing barriers to do it.

Is hiring hard?  Well, yes it is for everyone in the US.  Gosh.  What have they been doing for the last 5 years? 

My point is that the culture prevents them from starting today.  There was no barrier for them to start 5 years ago.  Stop every one of the standards groups, buy some VMWare licenses, Pick some Products and start coding.  Today.  Don't wait.  Do not do any product evaluations.  Don't bother with RFPs.  Just go.  Many reasons can be invented to not have a group of services created (let's set a goal of 200 new services per company per year).  But the real reason is culture. 

To be specific about the cultural issue, it has to do with accountability.  The model used in Telcos is that nobody can be fired for screwing up except at the bottom.  Committtees make decisions.  Those committees do detailed studies.  They debate fine points.  Nobody can be blamed for a bad decision.  The accountability is too diluted.  The growth portion of the product life cycle of these services is shoter than the decision cycle in a carrier.

Now, the question to me is why do SPs bother with this when it is like Johnny Cochran said, "If it doesn't fit, you must acquit."

seven

 
mhhf1ve
50%
50%
mhhf1ve,
User Rank: Light Sabre
4/24/2017 | 7:14:23 PM
Re: Diagnosis
> "The model used in Telcos is that nobody can be fired for screwing up except at the bottom.  Committtees make decisions.  Those committees do detailed studies.  They debate fine points.  Nobody can be blamed for a bad decision.  The accountability is too diluted."

That's the stick.. Where's the carrot? If smaller telcos had the incentives to actually try to do new things (ie. there was an upside in sight), there might be organizations that could avoid committee-decisions and just go. But when municipal telcos can't even get off the ground, and smaller telcos are struggling to cobble together contiguous networks... perhaps there's a larger problem at hand?
creynolds32701
50%
50%
creynolds32701,
User Rank: Light Beer
4/21/2017 | 9:22:05 AM
Re: Diagnosis
Hi Carol - Please reach out to me. I want your advice on some new products we are introducing to help in this area. 

thanks

 

www.cloudasafile.com

 
Steve Saunders
100%
0%
Steve Saunders,
User Rank: Blogger
4/20/2017 | 9:38:49 AM
Re: Diagnosis
"As for standards bodies, there is much greater engagement with open source groups today - every major operator I know is sending developers to groups like OpenDaylight, ONAP, TIP and more. "

 

The service providers i talk to are absolutely furious about the number and variety of open source groups that they have to engage with. I totally agree. It's jobs for the geeks. 

Self perpetuating geek oligarchy. 

Reminds me of  the high speed LAN wars of the nineties. Bunch of crap. Pick one guys, and then get a real job.  

 
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/19/2017 | 4:31:42 PM
Re: Diagnosis
but you can introduce as many services as you like when you like if they are deployed as interoperable VNFs... 
James_B_Crawshaw
50%
50%
James_B_Crawshaw,
User Rank: Blogger
4/19/2017 | 4:19:56 PM
Re: Diagnosis
I'm pretty sure the earth revolves around the sun not the other way round. Otherwise I think you are right about the pace of change being too slow. It's hard to turn a supertanker though.
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/19/2017 | 4:33:30 PM
Re: Diagnosis
easier to turn the supertanker when it doesn't have superwankers driving it. 

By which i mean that a ot of people in our industry need to stop pretending they are working in a industry that doesn't actually exist.

CSPs need HELP to rebuild their businesses and, as an industry, we are failing utterly to provide the solutions that they need! 

We need to fix that. 

 

 

 

 
mendyk
50%
50%
mendyk,
User Rank: Light Sabre
4/19/2017 | 5:00:17 PM
Re: Diagnosis
CSPs also need to get their acts together, collectively and individually. At this point, for instance, Verizon is succeeding in spite of some pretty strange management decisions, none of which have anything to do with virtualization.  
Voluntee24126
100%
0%
Voluntee24126,
User Rank: Light Beer
4/21/2017 | 5:15:35 PM
Re: Diagnosis
The issue the Telcos have is that if Facebook or Google have an outage, it might hit the internet news if it is big enough.  A major outage by a Telco hits all of the news outlets and if it is a multi-region one, their CEO and/or CTO end up testifying before Congress.  The stakes are much higher for the Telcos than Facebook and Google.  Fault recognition and switch to protect time in the core telecom network is 51ms.  As you have noted, there are few specifications in the Facebook/Google world.  The ones that I have seen reported are 5 or more minutes for fault recognition.  I have run across no time limit specs for protection switching for the social media or even cloud vendors. 

The key here is competing with the OTT application vendors while maintainng the basic reliability of the core network.  Not an easy task.

 
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/19/2017 | 4:30:23 PM
Re: Diagnosis
the CSPs wanting to be like OTTs thing is just a retarded form of technology penis envy. 

I remember when i was working in one of the more stupid salt mines of UBM and was involved in a stategic committee to reinvent the business... an executive, who shall remain nameless (not reallly, it was David Berlind) suddenly shouted out "we must work out how we can be more like google??!" i mean, FFS, why does that make any sense at all.

CSPs are not google or facebook andnever will be ... they need to do better at sticking to 

  their knitting. 
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/19/2017 | 4:27:16 PM
Re: Diagnosis
seven - very well said and i concur. We're calling an orange a banana and feeding it to an angry monkey
Duh!
100%
0%
Duh!,
User Rank: Blogger
4/20/2017 | 4:23:38 PM
Re: Diagnosis
I also concur, with a few further points.

Cargo cults (look it up) are an apt metaphor for imagining that virtualization will make CSPs like Google et. al. A few of the trappings of the scale-out datacenter are necessary, but not sufficient. Taken out of their full context, they are risible. Kind of like those mock airports in Micronesia.

The cargo cult posits that massive cultural change can make it all work. I think they fail to realize depth of the culture change that is needed, and the institutional immune responses to be overcome.  Google's engineers think like computer scientists; CSP engineers think like telecom engineers. Ordinary minds will revert to the way of thinking they grew up with. Change isn't a simple matter of courseware and threats.

DevOps and CI/CD, for example. Facebook has no fear of pushing out new software, and if there are problems, rolling it back and fixing it in the next release, no big deal. How do you explain that to people for whom the blood/brain barrier between engineering and ops has always been a given? We don't need regression testing anymore? Field trials involving unproven code get conducted with live, not-necessarily-friendly customers?

Speaking of thinking like computer scientists, consider ETSI's NFV model. Lots of functional boxes, connected by a hodge-podge of numerous and disparate interfaces. Computer science notions like abstraction, recursion, remote procedure calls, and inheritance are apparently absent. Would or could such a model be drawn for a scale-out datacenter?

On the other hand, never underestimate the determination of this industry, once it's been sold a panacea, to make it work. They'll muddle along until it does.
briansoloducha
100%
0%
briansoloducha,
User Rank: Moderator
4/20/2017 | 4:40:47 PM
Re: Diagnosis
"Facebook has no fear of pushing out new software, and if there are problems, rolling it back and fixing it in the next release, no big deal. How do you explain that to people for whom the blood/brain barrier between engineering and ops has always been a given?"

Better question - how do you try and bring that into your company when your company has a history of FIRING PEOPLE who do this? We need approvals up to VP-level to do changes planned 3-weeks in advance for a mid-week, middle-of-the-night change. If people circumvent this approval process, they do so at risk of immediate dismissal. And yet, management here wants to change the culture....

Thank you for discussing this topic.
linkedin54492
100%
0%
linkedin54492,
User Rank: Light Beer
4/19/2017 | 9:10:57 PM
There's another way to look at NFV...
If you look at NFV as a decade old strategic sales development effort by the Intel x86 marketing team, executed extraordinarily well through many points of action in the industry, the current state (which is certainly not business as usual in the telco industry) is not a disaster, it's just an intermediate state through which one would expect a change of this magnitude to progress.

The industry is simply sorting out what's off the shelf commodity (x86, NICs, and virtualization), what's purpose built for telco (the VFs), and what is curated commodity (like what Red Hat does for Linux).  The hardware is clear (at least for the moment) and it is in the software space that the telco goal of using COTS/commodity for CAPEX reasons meets the various-vendor goal of getting additional gross margin from purpose-building or at least curating.  

May the marketplace reach a suitable outcome, in the true spirit of capitalism!
Distingu57543
50%
50%
Distingu57543,
User Rank: Light Beer
4/20/2017 | 12:11:24 AM
The Cloud IS the Commodity
Have always thought the industry published a decent NFV white paper in October of 2012 and then failed to follow its own plan. The white paper talked about leveraging the capabilities of the hyper-scale IT cloud platform operators but then the industry decided to reinvent the cloud platform.

The industry lost track of the distinct separation that needs to be maintained between Tenant VNF / Service Orchestration Layer and Platform Resource Management / Orchestration Layer. As a result, Network Function vendors focused on solutions that encompassed both their version of a cloud platform and their version of Software. This was exacerbated by the absence of standards. The resulting VNF software was not truly "cloud aware" and not portable across even more mature "commodity cloud" platforms. This becomes a huge problem in a digital era when virtually all content delivery is a multi-cloud, multi-service provider endeavor.

All Network Functions are not the same. They do not impose the same low-latency and networking requirements. The industry could have chosen to start with those network functions most easily re-architected to be cloud aware so as to run optimally as SaaS on a multi-tenant "commodity cloud". The more difficult, low-latency, Control Plane VNFs could have been addressed in a second or third wave. Not too late to make this course correction.

As for Cloud Platforms, the closed loop, AI enhanced, data analytics driven, software defined, resource graph/state management capabilities of a globally distributed cloud platform like Microsoft Azure significantly exceed the vision of ETSI MANO or Open MANO. The TM Forum has attempted to fill gaps with the ZOOM initiative and OSS Future Mode of Operations work but even that has difficulty keeping up with the accelerating advancement of hyper-scale cloud platforms and the latest software architecture and DevOps trends. 

The telecom NFV ecosystem focus on commodity x86 hardware and combining it with Open Source software got everyone off track. It was fully engineered and commercially stable Cloud Platforms that were becoming the actual commodity. The hyper-scale cloud developers and operators were far more advanced with VIM and NFVI than the telecom industry realized. They had already abstracted hardware lifecycle management from software lifecycle management to create apparent unlimited scalability. This may have been difficult to recognize in 2012 but it is becoming very clear in 2017.
Foundera31843
50%
50%
Foundera31843,
User Rank: Light Beer
4/20/2017 | 3:18:41 AM
What is a service?
Once you compare the likes of the OTT's and the SP and their ability to use or drive value from DevOps a few mistakes are often seen: 

 

1) OTT is targeting the whole world, not just possible customers that are tied to a speciific access network. This means scaling matters and consumer uptake is from a much wider pool of possible subscribers. This is a matter of risk managment when a new service is introduced. 

2) OTT does not generally charge the end customer for the service but rely on ad-funding. This means that charging subsystems are much different and scale of the complexity is much different. e.g. this is a significant Opex and Capex different. 

3) As OTT generally does not charge the end customer expectations of customer support etc is very different. This is significant in means of Opex. 

4) A OTT can in many cases build their full stack them self as this is relatively self-confined. It is at least a order of magnitude less complex to build for example a video server compared to a HSS or a MME or a RNC. The amount of regression testing required on a telco node compared to a OTT node is very very significant, even with CI/CD in a automated manner this process take a long time to execute and once you move into a multi vendor scenario things get even more complex.

Does it make sense to even consider daily relesaes of a RAN or a EPC or is this without any value? Should the focus be on the specific services instead? 

5) Many people seem to have forgotten that NTT DoCoMo to a large extent have managed their own internal SW development on top of a set of HW provided by their family vendors. If we want to evaluate the cost of internal SW development in a SP environment I believeNTTis the most appropriate place to start to benchmark. 

6) WAP was the 1990 leap of fait for service revenue, it failed in all markets where the operator did not choose to embrace revenue share and co-development/risk sharing models, i-Mode was the only one (that I'm aware off) that both offered a set of useful and attractive services and generated revenue. 

7) Some of the industry groups are now looking at how we can target a zero-touch network management model that would significantly offset the Opex requirements. The work is just starting but it is clear that the target is not to improve just 10 or 15% but to go to a whole new model with zero-touch. This is similar to how SON was developed to eliminate certain very time consuming tasks in RAN planning and basic optimization. The challenge is to see to what extent this needs and can be deployed in hybrid scenarios vs. only in green-field deployments with all new interfaces and capabilities. 

So with this in mind does it make sense for a SP to build services that they only offer to subscribers of their own access networks or should they go head2head with OTT players and design and build global services that might have certain extra value add on their own access networks? OR should the SP's offer a strong revenue share model and actually open up and provide further value to their partners in the eco system and via this de-risk and co-create services? 

 

 
Martin Morgan
50%
50%
Martin Morgan,
User Rank: Light Beer
4/20/2017 | 10:27:43 AM
Virtualisation and vendor lock-in at BSS level
Virtualisation is supposed to deliver a more open ecosystem that allows operators to use best of breed solutions and steer clear of vendor lock in. At the OSS and BSS level many vendors are adopting virtualisation to enable this – while some are trying to maintain lock-in and supply everything to operators. On the BSS side there's more than one way to get away from vendor lock in. What we've seen is the use of adjunct systems, e.g. real-time OCS implemented in front of legacy billing systems, work well. Most specialist vendors want open and easily interoperable systems – any industry advances against vendor lock in closed shops are to be welcomed.
Frodo_ALU
50%
50%
Frodo_ALU,
User Rank: Light Beer
4/20/2017 | 12:59:53 PM
History repeating?
Like the boy seeing the naked emperor, I can't shake the impression that SDN/NFV is, in the most general sense, another management interface standard.  How many times have we tried and failed to truly standardize the management interface?  We get most of the way there, but there's always some indispensible needs that are not standardized, that prevent true interoperability.  We despair at the last high hurdles before the finish line.  We start over from scratch, with unrealistic confidence that there will be no last high hurdles this time around.
saints4454
100%
0%
saints4454,
User Rank: Lightning
4/20/2017 | 2:03:20 PM
Virtualize or Automate?
I really liked your article and agree with many aspects.  For the past few years, I've stressed the need for investment more so in Automation of Telecom networks vs. the virtualization of network functions.

Dramatic efficiencies in Operations can be realized by starting with transforming how SP's operate their networks.  

Over the past 4 years, if 50% of the virtualization investments would have been made in automation initiatives, I believe that the SP's capabilities to adopt virutalization would be far better and they would be in a better position to compete with the innovation being seen from OTT players.

Automate and Augment first.

Brad
Steve Saunders
50%
50%
Steve Saunders,
User Rank: Blogger
4/20/2017 | 2:43:20 PM
Re: Virtualize or Automate?
very interesting point, Brad!
Carol Wilson
50%
50%
Carol Wilson,
User Rank: Blogger
4/20/2017 | 3:09:18 PM
Re: Virtualize or Automate?
Brad,

Your comment is perfectly in line with what every major service provider says. It is the reason the Open Network Automation Platform project exists. 

 

Carol 
saints4454
50%
50%
saints4454,
User Rank: Lightning
4/20/2017 | 4:20:15 PM
Re: Virtualize or Automate?
I agree Carol.  ONAP definitely has the potential to be beneficial for SP's.

 

Brad
arifhrashid
100%
0%
arifhrashid,
User Rank: Light Beer
4/21/2017 | 1:40:46 AM
Time for Telecom Reboot
True , not a simple reboot but a deeplevel reformat is required.
creynolds32701
50%
50%
creynolds32701,
User Rank: Light Beer
4/21/2017 | 9:17:45 AM
re: Help with your initiative
Hi Stephen - Please reach out to me. We have some technology that may help with the standards and getting everyone to share and use the IP developed. 

Chuck Reynolds

TSI 

www.cloudasafile.com

 
reimagine_networking
50%
50%
reimagine_networking,
User Rank: Blogger
4/25/2017 | 9:49:42 AM
Its Not Virtualization -- but the Foundation
You are correct that we need standards for VNF's to inter-operate, but the problems are in the routing plane. We are relying on tunnels, ethernet and broadcast domains for security (L2) instead of L3. We need to rethink how IP networking works. The correct foundation for IP services is not L2, but rather L3. VLANs are not routable, are not accessible at the application layer, and do not scale. VxLANs are advisory, and utilize waseful encapuslation. I agree there should be a call for standards. Lets go back to 1993, and reimagine how IP routing could and should work. Lets end layer violations by always resorting to ethernet solutions.
Duh!
50%
50%
Duh!,
User Rank: Blogger
4/27/2017 | 6:57:21 PM
Re: Its Not Virtualization -- but the Foundation
"Lets go back to 1993, and reimagine how IP routing could and should work."

We did.  It was called ATM. The Betamax of networking technologies.
brooks7
50%
50%
brooks7,
User Rank: Light Sabre
4/28/2017 | 12:17:34 PM
Re: Its Not Virtualization -- but the Foundation
@Duh!,

Good one. 

seven
rommelb
50%
50%
rommelb,
User Rank: Light Beer
4/25/2017 | 2:42:24 PM
Interesting article, but....
Disclaimer: This is my view.  I am constantly learning, and would love to understand other thoughts.

Interesting article, and I do have to agree there is a culture mismatch  But there are things that can be done today.  Just like ATT in its early inception, google had a journey as well.  ATT built analog switches to move away from cord boards, because there was nothing there to help them grow and optimize.  Just as the OTT players had to use software to build more efficiently optimized networks and services because there was nothing there to shore up their growth.

Building blocks for changing how to optimize business and operational process are there, just as interface standards are there for interop.  Can it be better? yes, but there are areas to start.  Boiling the ocean of orchestration across a full customer facing service stack is difficult if the small successes of automating anything done twice, hasnt been built as foundation on the infrastructure. Automated outage recovery is difficult without understanding how the network is modeled and an understanding of types of outages and blast radius of those types.  Abstracting and automation of resource facing services on the network both PNF and VNF has to be done in order to orchestrate the NF relationships.  This all has to be done before one can even activate services on that infrastructure for customer consumption

My bottom line recommendation; Understand the processes, risks and impacts, automate everything, abstract the information so that automation is reusable,  orchestrate lifecycle where it makes sense, understand your data and optimize with that data using the automated tools built.  A meal like that is best taken in bite sized chunks.  

Start with training courses on software like Ansible, maybe get programs in place like automation of the month or quarter for the network engineers that create something with some gains.  The simple win of creating a culture of automation and optimization can lead to operations efficiencies that can pay back.
scanlanavia
50%
50%
scanlanavia,
User Rank: Lightning
4/26/2017 | 9:42:41 AM
Frustrated Analyst .. there
No doubt the author fell out the wrong side of the bed when this was written.

However I tend to agree with most of what was said. Four years down the road and nothing much achieved except plethoras of white papers and marketing hype.

Can you tell me what on earth is the issue with poor old HPE .. They are not alone in stonewalling the exercises.

DevOps, DevOps. Devops..... this is the mantra  .. being showered at CSPs.

I agree entirely CSPs will NEVER successully put Devops in place... it's like the promise of the Agile Manifesto which likewise never worked for large organisations.. It's fine if you're making a small consumer app... But you cannot scale DevOPS likewise and hope that the daily update is going to smoothly avoid the occasional HLR meltdown for instance once in a while..  ( a catastrophe in otherwords )

CSPs cannot afford happy clappy failures..

I actually thing ETSI/NFV got off to a good start with their NFV showcases , usecases and proof of concepts...... but seemed to run out of inertia to finish the job off..

The Open Source is a like a  hamper of shiny chocolote wrappers each offering their own distinct piece of the jig saw. However the ecosystem is now so complex that even the PHD's can't quite figure out where to start anymore..

Well done for waking up the industry a bit... but I think it will take more than this to steer the industry from going over the cliff...
gregw33
50%
50%
gregw33,
User Rank: Lightning
4/26/2017 | 12:36:26 PM
Gaming Open Source and SDO's
Standards Development Organizations (De Jure and De Facto) are extremely easy to "game" ...

I hope readers don't think Open Source Software initiatives cannot be "gamed" as well...

They are...

GW

 
Kevin Mitchell
100%
0%
Kevin Mitchell,
User Rank: Light Sabre
4/26/2017 | 9:21:54 PM
Focus on Business Outcomes in Choosing Your Virtualization Path
If the goal is more revenue and lower cost by implementing "the best and most popular (and most profitable) mix of services by picking and choosing from an online marketplace chock full of 'best in class' services and applications", then buying NFVI this and integrating VNF that isn't necessarily the way to do it. 

We've been saying for years that operators need to look at business outcomes and evaluate the various paths to virtualization. Cloud building or cloud sourcing are the choices; look at the application in question and which path best maximizes the chance at realizing those outcomes. In many cases it will make sense to build a new virtualized network to support multiple applications.

Given the state and future direction of voice and UC we believe that, for the vast majority of operators, that cloud sourcing is the best way forward (yes, the global top 100 will build an NFV IMS network and complete that in 2023 or thereabouts). Why cloud source virtualized VoIP? It's lower cost (and no CAPEX) with a success-based business model, operationally simple, and it's tremendously most agile than software on premises. Oh, and CSPs get to spend dollars and people focus on other initiatives while still owning and delivering a modern communications services for its customers.

Yes, this CSP VoIP cloud sourcing is what we do (and first to do it). But the traditional VoIP players are getting into this game too (BroadSoft BroadCloud is the fastest growing part of its business, GENBAND is a KANDY junkie and Metaswitch has joined the club).

A Heavy Reading study confirmed this too: 83% of CSP respondents said that it was somewhat or very likely that they'd use a XaaS option for replacing or augmenting network infrastructure. And, along with building VoIP, cloud voice platforms was a top voice network evolution path. Read more here -> Heavy Reading: Cloud Defines a New Voice Strategy.

 

 
More Blogs from From the Founder
After almost two decades at Light Reading, it's time for a different optical adventure.
John Chambers is still as passionate about business and innovation as he ever was at Cisco, finds Steve Saunders.
Light Reading founder Steve Saunders talks with VMware's Shekar Ayyar, who explains why cloud architectures are becoming more distributed, what that means for workloads, and why telcos can still be significant cloud services players.
Light Reading's recent Automation Everywhere conference provided invaluable guidance and insights for network operators figuring out their automation strategies.
Ngena's global 'network of networks' solves a problem that the telecom vendors promised us would never exist. That doesn't mean its new service isn't a really good idea.
Featured Video
Flash Poll
Upcoming Live Events
September 17-19, 2019, Dallas, Texas
October 1, 2019, New Orleans, Louisiana
October 2-22, 2019, Los Angeles, CA
October 10, 2019, New York, New York
November 5, 2019, London, England
November 7, 2019, London, UK
December 3-5, 2019, Vienna, Austria
December 3, 2019, New York, New York
All Upcoming Live Events
Partner Perspectives - content from our sponsors
Transform Beyond Borders to Lead the Innovation
By Ben Zhou, CEO, Whale Cloud
Reject Limits. Build the Future.
By David Wang, Huawei
China Telecom & Huawei Jointly Complete the World's First End-to-End 5G SA Voice & Video Call
By Jay Liu, Senior Marketing Manager, Cloud Core Product Line, Huawei Technologies
All Partner Perspectives