& cplSiteName &

Facebook: Yes, We Need 100-GigE

Craig Matsumoto
News Analysis
Craig Matsumoto
9/16/2009
50%
50%

It's become cliché to say that companies like Facebook would use 100-Gbit/s Ethernet right now if they had it. But it helps when someone from Facebook actually shows up and hammers on that point.

Facebook network engineer Donn Lee did that yesterday, pleading his case at a technology seminar on 40- and 100-Gbit/s Ethernet, hosted in Santa Clara, Calif., by The Ethernet Alliance .

Representatives from Google (Nasdaq: GOOG) and the Amsterdam Internet Exchange B.V. (AMS-IX) gave similar pleas, but Lee's presentation included some particularly sobering numbers. He said it's reasonable to think Facebook will need its data center backbone fabric to grow to 64 Tbit/s total capacity by the end of next year.

How to build such a thing? Lee said his ideal Ethernet box would have 16-Tbit/s switching capacity and 80 100-Gbit/s Ethernet ports or 800 10-Gbit/s Ethernet ports.

No such box exists commercially, and Lee is reluctant to go build his own.

That leaves him with an unpleasant alternative. Lee drew up a diagram of what Facebook's future data center fabric -- that is, the interconnection of its switch/routers -- would look like if he had to use today's equipment and 10-Gbit/s Ethernet. Instead of the familiar criss-crossing mesh diagram, he got a solid wall of black, signifying just how many connections he'd need.

"I would say anybody in the top 25 Websites easily has this problem," he said later. (Lee didn't say anything about how long it would take just to plug in all those fibers. Maybe that job could be created by funds from the U.S. Recovery Act.)

Lee also showed charts showing the disconnect between Facebook's wish list and the market. Facebook needed 512 10-Gbit/s Ethernet ports per chassis in 2007 and is likely to need 1,500 in 2010. No chassis offers more than 200 ports, he said.

Even though Lee is a veteran of Google and Cisco Systems Inc. (Nasdaq: CSCO), you might wonder if he's just one renegade engineer who doesn't represent the Facebook norm. Not really. It turns out Facebook has only five network engineers -- although Lee said that's a 20 percent increase from the spring of 2008 [math note: which means they had approximately 4.165 engineers at that time].

Even though 100-Gbit/s development started four years ago, Lee thinks it came too late, and that's got him worried about the next generation. He's pulling for 400-Gbit/s Ethernet discussions to start right away.

"Let's start the work that doesn't require money, now," he said. "If we have the standard, we can build the product later. I don't mind using an old standard."

He might get his wish. The Optoelectronics Industry Development Association (OIDA) is already organizing meetings with an aim toward getting federal money for terabit Ethernet research, said John D'Ambrosia, a Force10 Networks Inc. scientist who helped organize yesterday's event.

Of course, money is a major obstacle to the next wave of Ethernet.

During an open commenting and Q&A session, multiple audience members pointed out that optical components margins are too thin to support advanced research at many companies, and that carriers are seeing their big, expensive networks getting used to make money for over-the-top services. "There's no revenue in all that bandwidth increase," one audience member commented, citing the carrier case in particular.

— Craig Matsumoto, West Coast Editor, Light Reading

(15)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Page 1 / 2   >   >>
Pete Baldwin
50%
50%
Pete Baldwin,
User Rank: Light Beer
12/5/2012 | 3:56:22 PM
re: Facebook: Yes, We Need 100-GigE


 

Lee presented graphs of how Facebook traffic can surge at certain times. The audience got to guess what the causes were.

Here are a few of his examples (purely from memory, so they don't have five-9s accuracy).  Take your best guesses. No fair if you were at the meeting, or if you work for a social networking site!

1. Nov. 1, 2008, most of the morning.

2. Feb. 1, 2009, sporadic spikes throughout the evening (Eastern time)

3. June 25, 2009, one sudden, enormous burst, trailing off hyperbolically

4. Entire week of Aug. 31, 2009 -- traffic consistently higher than the previous week, for 24 hours every day (but retaining the same shape, just shifted upwards)

ninjaturtle
50%
50%
ninjaturtle,
User Rank: Light Sabre
12/5/2012 | 3:56:21 PM
re: Facebook: Yes, We Need 100-GigE
INFN to the rescue...sorry couldn't resist. Google already is a customer. Is it feasible to "Kevin Bacon" that to INFN also???? Ka-ching!
TrojanReal
50%
50%
TrojanReal,
User Rank: Light Beer
12/5/2012 | 3:56:20 PM
re: Facebook: Yes, We Need 100-GigE


" Lee drew up a diagram of what Facebook's future data center fabric -- that is, the interconnection of its switch/routers -- would look like if he had to use today's equipment and 10-Gbit/s Ethernet. Instead of the familiar criss-crossing mesh diagram, he got a solid wall of black, signifying just how many connections he'd need."

Can someone clarify, if the  problem described above is primarily due to   having fixed optical connections  between sites  (as opposed to dynamic  optical connection with timescales  of  nsec,usec duration @ 10Gbps as in optical packet or burst switching)? Or  if it is due to each connection requiring 100G bandwidth.? 

Thanks
TR

chaz6
50%
50%
chaz6,
User Rank: Light Beer
12/5/2012 | 3:56:20 PM
re: Facebook: Yes, We Need 100-GigE


I do wonder why they desperately need to use ethernet. Take for example the Mellanox IS5600; it has 648 ports of 40Gbs with a switching capacity of 51.8 Tb/s, far exceeding the number bandied about by Facebook.

savy.tech
50%
50%
savy.tech,
User Rank: Light Beer
12/5/2012 | 3:56:19 PM
re: Facebook: Yes, We Need 100-GigE


Answer to your puzzle


------------------------


June 25, 2009, one sudden, enormous burst, trailing off hyperbolically


Death of Michael Jackson caused the traffic spikes on most of the websites including Twitter and other news sites..


abashford
50%
50%
abashford,
User Rank: Light Beer
12/5/2012 | 3:56:16 PM
re: Facebook: Yes, We Need 100-GigE


"No such box exists commercially, and Lee is reluctant to go build his own."


They could get a lot more interest from manufacturers if they could indicate to the market that more than 2 of these 'boxes' are required. :)


I have no knowledge of Facebook's architecture, does anyone know how many major data centres they maintain?  Is it 2, or are they more distributed?  Thanks.

Pete Baldwin
50%
50%
Pete Baldwin,
User Rank: Light Beer
12/5/2012 | 3:56:13 PM
re: Facebook: Yes, We Need 100-GigE


abashford -- He'd need 16 of them, if I undertand correctly.  And more in the future, of course -- but not thousands of them, admittedly.


During the Q&A, someone actually brought up your point about volumes, and that's when Lee made the "top 25 web sites" remark, IIRC.  He believes there's a hungry high-end market.


(Google apparently made similar comments earlier in the day, but I wasn't present for that session.  People wanted to bring him into the Facebook discussion, but he'd already left.)

Pete Baldwin
50%
50%
Pete Baldwin,
User Rank: Light Beer
12/5/2012 | 3:56:13 PM
re: Facebook: Yes, We Need 100-GigE


TrojanReal -- It's due to needing 100 Gbit/s bandwidth.  He also said there are spots where they'd be link-aggregating 100GEs, if they could.

abashford
50%
50%
abashford,
User Rank: Light Beer
12/5/2012 | 3:56:12 PM
re: Facebook: Yes, We Need 100-GigE


Sounds like a large investment for a potentially small market.  But then, that is what people were saying about the CRS-1 before it sold in the Nx100's.  If you are the only option, it could be a lucrative market.


It would seem more efficient to actually come up with a cost-effective and simple way to solve the meshing of multiple smaller devices to look like one large one.  That way, you can invest in the development of a device that has a much larger target market.  I have yet to see anyone pull this off however...


Was there any discussion of potential technologies to do this?


 

Pete Baldwin
50%
50%
Pete Baldwin,
User Rank: Light Beer
12/5/2012 | 3:56:12 PM
re: Facebook: Yes, We Need 100-GigE


Ah - the "20 percent increase" cited should be 25 percent.  That's my math mistake, not Lee's (the parenthetical was added in edit).


I'm losing my math chops.  I blame Twitter.

Page 1 / 2   >   >>
Educational Resources
sponsor supplied content
Educational Resources Archive
Light Reading’s Upskill U is a FREE, interactive, online educational resource that delivers must-have education on themes that relate to the overall business transformation taking place in the communications industry.
NEXT COURSE
Wednesday, September 28, 1:00PM EDT
Gigabit 101
Will Barkis, Senior Technology Analyst, Orange
UPCOMING COURSE SCHEDULE
Friday, September 30, 1:00PM EDT
Gigabit & the Great Migration
Robert Howald, Vice President, Network Architecture, Comcast
Wednesday, October 5, 1:00PM EDT
Gigabit & Smart Cities
Joe Kochan, COO & Co-Founder, US Ignite
Friday, October 7, 1:00PM EDT
Gigabit & DOCSIS 3.1
Ty Pearman, Director, Access Architecture, Comcast
in association with:
From The Founder
Light Reading today starts a new voyage as part of a larger Enterprise.
Flash Poll
Live Streaming Video
Charting the CSP's Future
Six different communications service providers join to debate their visions of the future CSP, following a landmark presentation from AT&T on its massive virtualization efforts and a look back on where the telecom industry has been and where it's going from two industry veterans.
LRTV Documentaries
MEC Congress: The Key Takeaways

9|22|16   |   03:25   |   (3) comments


Three key takeaways from the Mobile Edge Computing (MEC) Congress in Munich, Germany.
Wagner’s Ring
Time to Shut Up About 'Dumb Pipes'

9|22|16   |     |   (12) comments


Service providers can't compete with OTT players. It just isn't in their DNA. Instead, service providers need to embrace what they're good at -- providing reliable, secure connectivity.
Wagner’s Ring
Keeping Your Tech Career Going After 50

9|21|16   |     |   (13) comments


How do you keep your career moving forward when you're past the half-century mark?
LRTV Interviews
Peering Into the Digital Future

9|20|16   |   04:25   |   (0) comments


Nick Thomas, practice leader of digital media at Ovum, talks about how digital transformation in the technology, media and telecom sectors will enable the development of a new range of applications and services for enterprises and consumers and how the upcoming Digital Futures event in London will examine ...
LRTV Custom TV
Napatech Tackles NFV's Major Challenge

9|7|16   |   08:42   |   (0) comments


One of the main challenges for network operators introducing NFV is to combine performance and flexibility in a cost-effective way, but there is a solution, explains Napatech's Dan Joe Barry.
LRTV Custom TV
Huawei Optical Innovation 2016

9|6|16   |     |   (0) comments


Highlights of the 2016 Huawei Optical Innovation forum.
LRTV Huawei Video Resource Center
Are You Ready for Huawei Connect 2016?

8|31|16   |     |   (0) comments


Join us for an exclusive sneak peak of Huawei Connect, an integrated conference for the global ICT ecosystem taking place in Shanghai.
Between the CEOs
CEO Chat: UXP's Gemini Waghmare

8|26|16   |     |   (0) comments


Light Reading CEO Steve Saunders and UXP Systems CEO Gemini Waghmare discuss the strategic importance of digital identity for operators in the midst of transformation.
LRTV Custom TV
F5 Virtual Network Function Integrations With Partner Orchestration Platform

8|24|16   |   6:38   |   (0) comments


F5's Kishore Patnam, product manager for F5's service provider solutions, discusses why service providers are moving towards virtualization and how his clients are utilizing F5's solutions.
Between the CEOs
CEO Chat: Intel's Alexis Black Bjorlin

8|17|16   |   06:23   |   (0) comments


Join us for an in-depth interview between Steve Saunders of Light Reading and Alexis Black Bjorlin of Intel as they discuss the release of the company's Silicon Photonics platform, its performance, long-term prospects, customer expectations and much more.
Telecom Innovators Video Showcase
Accelerating Telecom Digital Transformation With Nominum DNS

8|1|16   |   12:04   |   (0) comments


Light Reading's Steve Saunders gets an update from Nominum CEO Gary Messiana on how his company is helping carriers on the digital transformation journey.
LRTV Custom TV
Reinventing Operations for a Virtual, Software-Defined World

7|28|16   |   5:23   |   (0) comments


Heavy Reading Senior Analyst Jim Hodges speaks with Accenture's Larry Socher and Matt Anderson about what service providers must do to transform their business to get the benefits of SDN and NFV including: leveraging DevOps, introducing real-time OSS and implementing analytics.
Upcoming Live Events
November 3, 2016, The Montcalm Marble Arch, London
November 30, 2016, The Westin Times Square, New York City
December 1, 2016, The Westin Times Square, New York, NY
December 6-8, 2016, The Westin Excelsior, Rome
May 16-17, 2017, Austin Convention Center, Austin, TX
All Upcoming Live Events
Infographics
Hot Topics
Keeping Your Tech Career Going After 50
Mitch Wagner, West Coast Bureau Chief, Light Reading, 9/21/2016
Time to Shut Up About 'Dumb Pipes'
Mitch Wagner, West Coast Bureau Chief, Light Reading, 9/22/2016
Comcast Will Go Wireless in 2017
Mari Silbey, Senior Editor, Cable/Video, 9/20/2016
Sprint CEO: We'll Lease Spectrum as Soon as Possible
Dan Jones, Mobile Editor, 9/20/2016
Like Us on Facebook
Twitter Feed
BETWEEN THE CEOs - Executive Interviews
Light Reading CEO Steve Saunders and UXP Systems CEO Gemini Waghmare discuss the strategic importance of digital identity for operators in the midst of transformation.
Join us for an in-depth interview between Steve Saunders of Light Reading and Alexis Black Bjorlin of Intel as they discuss the release of the company's Silicon Photonics platform, its performance, long-term prospects, customer expectations and much more.
Animals with Phones
You Thought Your Customer Service Was Unreliable... Click Here
Live Digital Audio

A vital part of increasing the number of women in comms is transforming the ways companies can support and empower women. While progressive company policies that support both men and women in achieving work-life balance are a step in the right direction, creating a company culture that supports those policies can at times be more challenging.

During this show, we'll talk to Lynn Comp, Senior Director of Industry and Sales Enabling (ISE) in the Network Platforms Group at Intel, about why those challenges exist and how companies can overcome them. She'll provide insight into how Intel has worked to create a culture that supports work-life balance, and provide steps and guidance for other companies wishing to do the same. We will also leave plenty of time to get your questions answered live on the air.