OpenStack Underpins Comcast Elastic Cloud

Major telecom network operators are not the only ones stepping up to proclaim their support of OpenStack, this open source cloud platform that is sometimes disparaged as a carrier platform.

Comcast Corp. (Nasdaq: CMCSA, CMCSK), the largest US cable operator, is actually also one of the larger OpenStack users in the networking segment, having deployed OpenStack in its national data centers and some of its regional ones as well. In fact, if you use Comcast residential email, you may already be served by an OpenStack platform. (See Showdown at the OpenStack Corral, Verizon NFV Plan Pushes OpenStack Forward and AT&T Rallies Carriers Around OpenStack.)

Comcast began investing in OpenStack in 2012 as a member of the OpenStack Foundation and has built its Comcast Elastic Cloud on the open source standard. Today it has 27 OpenStack environments scattered around the US, the majority of which are at the edge of the network, says Mark Muehl, Comcast's SVP Platform Technologies.

The buildout is part of Comcast's smarter network plan, which creates "a fabric of compute storage and connectivity in a cyber-security wrapper," and is designed to let Comcast move beyond configuring its network to programming that network.

"Our job is to generate options for our company so we can provide the choices that enable the best possible experience across a wide variety of applications that serve both residential and business customers through this smarter network," Muehl says. "Today we configure our network -- it's a pretty classic ISP network with massive scale. It is converged -- we run five billion-dollar businesses on top of the same infrastructure. But we have reached the scale where we need to do more than configure the network, we needed to program the network."

Learn more about the future of open source in telecom at our upcoming Big Communications Event in Austin, TX, May 24-25. Register now!

OpenStack plays a critical role in enabling this as the core platform and one Comcast has been developing for some time now. Working at a scale that delivers petabytes of memory for use by applications and millions of CPU cores, the cable MSO -- like its telecom network competitors -- is moving off big iron to a software-based distributed architecture that will run on its OpenStack environments.

Where those sit is based entirely on the needs of the application, whether it is latency sensitive or generates traffic volumes that shouldn't be backhauled into the core, or whether it needs a centralized location.

Weighing options
As the folks in charge of giving Comcast options, Muehl and the Platform Technologies crew looked at their own options for creating a programmable network and decided the open source approach was better than building it all themselves or relying on a third party to build it for them, he says. The latter approach left Comcast too tied to someone else's timetable and at the mercy of their mistakes.

"We use open source in a variety of ways around our network and OpenStack is one of the premier open source projects that we work on -- and we are very engaged in the OpenStack community," he notes.

Comcast has contributed about 55,000 lines of code and has 611 commits since 2013, Muehl says. "For the last four releases of OpenStack, Comcast has been a top 20 contributor when you look at total reviews and commits," he adds. "We are very active in this community and this is one of the places where the Comcast team is well-received."

Comcast received the 2015 Superuser award from the OpenStack community -- something AT&T Inc. (NYSE: T) claimed this year -- in recognition of the ways it is using OpenStack but also the ways it is contributing back into the community.

And now Comcast, which has been using the Icehouse release for most of its installations, will be making the jump to Mitaka, the current stable OpenStack release as of April.

OpenStack evolution
In the four years that Comcast has been working in the OpenStack Foundation and its projects, Muehl says there are been major improvements in scalability, reliability and stability. On the latter front, Comcast has been able to move from doing OpenStack in regional deployments for 25 to 50 compute nodes to doing hundreds of compute nodes per region and developing a design, not yet implemented for 1,000 compute nodes per region.

"If you are a user of this platform, one of the characteristics you are looking for is the illusion of infinite capacity," he says. "You want to be able to go into 'the cloud' and scale up and scale down without having to think about whether the underlying infrastructure beneath you can handle your demands."

Comcast has evolved from using OpenStack on an experimental basis with only its "most capable" development teams -- such as the crew that worked on its X1 video platform -- to moving a wider variety of applications onto OpenStack as its reliability and stability improved. Its residential email platform, once deployed as vertical integrated applications, is now being broken into components and moved onto an OpenStack deployment using virtual machines and storage.

The cable operator is also using OpenStack to address issues such as dealing with the network telemetry data that comes from its routers, switches and other gear and needs to be analyzed to drive capacity planning and understand network performance. Typically, those duties would be executed on a Hadoop cluster and would require Comcast to invest in memory, storage and compute resources and hope it got the balance right.

Disaggregating storage, compute and memory into different independent pools that run on OpenStack lets the company "scale and use them independently so we don't have to get the balance right from day one on network telemetry," Muehl says.

A ways to go
"There are things like networking that have gotten better but still have a long way to go," Muehl admits. The introduction of Neutron as a system for managing networks and IP addresses helped considerably on that front.

"The vendor ecosystem has gotten much more robust, maybe too robust," the Comcast technologist notes. "Vendors understand the importance of having OpenStack support and so they are more aggressive around their parochial needs than you might want from an open source project."

John Leddy, VP of network strategy for Comcast, says the company would also like to see the network and data center infrastructure evolve more to look like a seamless operation for compute and storage.

Comcast works with other major network operators -- Muehl mentions AT&T and NTT Group (NYSE: NTT) as two -- and would look at the possibility of a large network operator group within OpenStack as AT&T has suggested, as long as it didn't create a forked version of the open source software.

The experience of working in a large open source group such as the OpenStack Foundation has been a new one for Comcast but Muehl thinks the company "has adjusted well" to getting down in the weeds of the underlying infrastructure, learning how to wield its influence and working compatibly alongside some of its biggest competitors.

He believes some large telecom network operators who disparage OpenStack may be limited by their own companies' view of the future.

"Sometimes it depends on the maturation of the carrier more than maturation of the platform," Muehl says. "I think companies need to recognize that what it means to be a carrier ten years from now is very different from what it means today or meant ten years ago."

— Carol Wilson, Editor-at-Large, Light Reading