Data Center Infrastructure

Google: 'Great' Data Center Networks Essential

SANTA CLARA, Calif. -- Open Networking Summit -- Delivery of computing power will depend on networking in the future, Amin Vahdat, fellow and technical lead for networking at Google (Nasdaq: GOOG), said at a keynote here Wednesday.

Compute platforms face coming bottlenecks that will choke performance, he says. The end of Moore's Law is here, or will be here in a few years, requiring change for computing and distributed computing. Storage capacity has increased hugely but I/O isn't keeping up.

That leaves networking as the driver of future growth.

"Networking is at an inflection point in driving next-generation computer architecture," Vahdat said. "What computing means will largely be dependent on our ability to build great networks over the years."

Google Man
Google takes advantage of the control it exerts inside its data centers to drive network efficiencies, says Amin Vahdat, a top engineer for the cloud company.
Google takes advantage of the control it exerts inside its data centers to drive network efficiencies, says Amin Vahdat, a top engineer for the cloud company.

Data center networking, in particular, is a special requirement for Google, Vahdat said. Google's services depend on its data centers.

"We build building-scale -- warehouse-scale -- computers. Just row after row of computers and row after row of storage," he said. "And you can't build a world-class distributed computing infrastructure without world-class networking infrastructure."

Google takes advantage of the control it can exert within its own data centers to drive efficiency and scalability, Vahdat said.

Last year at this conference, Vahdat described its Andromeda SDN infrastructure used to link data centers worldwide. (See Google, Microsoft Challenge Service Providers.)

Andromeda slices the physical network into VPNs for customers. It's complemented by Freedome, Google's campus-level interconnect network, and Jupiter, within the data center.

Data center networking has different advantages and problems when compared with sending traffic over the public Internet and wide-area networks. The chief distinction is that a data center is completely under the control of a single organization, unlike the "capital-I Internet," Vahdat said.

"You build it and you control it," he said.

Other distinctions: round-trip times are "tiny," the data center is "massively multipath," bandwidth is more plentiful and uniform. The network makes little use of buffering, and bandwidth needs are massive -- the aggregate bandwidth of the "entire Internet" is "in one building," Vahdat said.

When Google built out its data centers, it had special needs that could not be met with off-the-shelf equipment.

"We could not buy for any price a data center network that would meet the needs of our deployments," Vahdat said. Products were focused on individual boxes, with each box controlled by its own command line interface, making management an overwhelming problem. "The notion of managing a thousand boxes as if they were a thousand boxes didn't cut it."

Google leveraged three key principles: Using merchant silicon in Clos topologies with centralized control.

Google needs to keep up with explosive growth. Over six years, data center bandwidth needs have grown fifty-fold, outpacing Moore's Law. A typical network today might have 10,000 or more switches, with a quarter million links and 10 million routing rules, Vahdat said.

Find out more about key developments related to the systems and technologies deployed in data centers on Light Reading's data center infrastructure channel

Google implemented several generations of data center networking technology to meet its needs. The first generation, code-named Firehose 1.0, didn't see production. Google tried to integrate networking with servers, which didn't work out, Vahdat said.

Firehose 1.1 went live, using the same technology and traditional switches. Initially, Firehose 1.1 was deployed as a "bag-on-the-side," with servers connected both to previous generation cluster routers and the new generation. When Google became confident in the new network, it eliminated the old, Vahdat said.

Firehose was followed by Watchtower, then Pluto running at 10 Gbit/s; Saturn, the fourth generation, has 288 ports of 10 Gbit/s making up the edge aggregation block.

Finally, Jupiter leverages the latest generation of merchant silicon, with 80 Tbit/s bandwidth, hosting an SDN software stack using OpenFlow.

Facebook also relies on efficient data centers to drive its services, and has invested heavily in pushing open source hardware and software design. (See Facebook Reinvents Data Center Networking.)

More from the Open Networking Summit:

— Mitch Wagner, Circle me on Google+ Follow me on TwitterVisit my LinkedIn profileFollow me on Facebook, West Coast Bureau Chief, Light Reading. Got a tip about SDN or NFV? Send it to [email protected]

DHagar 6/18/2015 | 5:15:02 PM
Re: A good reminder about importance of networks mendyk, it really goes to the fundamentals of design that recognizes the need for networks to provide the functionality, scalability, and performance.  If Moore's law is reaching its limits, the configuration and new network designs are going to be needed to meet the "demand" side driven by growth in data, IoT, etc., of data centers. 

Google is smart, and building the "expertise" necessary to provide these networks.
mendyk 6/18/2015 | 9:46:33 AM
Re: A good reminder about importance of networks This is a huge point, Ray -- and one that gets lost in the whole "change or die" message that network operators receive on a daily basis. No doubt that technology improvements require significant disruptive changes to the way networks are built and run. But operators that make those changes have a clear path to long-term survival and success, even if they don't try to remake themselves into an Amazon-like operation (as if Amazon's inability to turn a profit is worth emulating).
[email protected] 6/18/2015 | 6:28:04 AM
A good reminder about importance of networks Ever since the 'cloud' became the big buzz, there were some that ALWAYS highlighted the importance of the networks that actually have to connect data centers and carry the traffic to and from users....

Joe Weinman said it time and time again when he was at AT&T. HE didn't just have impressive equations, he had practical sense. And in 2006 he was even describing a network plan that is essentially a software-defined WAN, as Light Reading reported:


AT&T is serious about building an on-demand global network to dynamically provision bandwidth and utility computing for corporate customers, says Weinman. This will also include dynamic pricing of services, such as you find in the airline industry.


Optical's Future Looks Brighter

Sign In