It's become cliché to say that companies like Facebook would use 100-Gbit/s Ethernet right now if they had it. But it helps when someone from Facebook actually shows up and hammers on that point.
Facebook network engineer Donn Lee did that yesterday, pleading his case at a technology seminar on 40- and 100-Gbit/s Ethernet, hosted in Santa Clara, Calif., by The Ethernet Alliance .
Representatives from Google (Nasdaq: GOOG) and the Amsterdam Internet Exchange B.V. (AMS-IX) gave similar pleas, but Lee's presentation included some particularly sobering numbers. He said it's reasonable to think Facebook will need its data center backbone fabric to grow to 64 Tbit/s total capacity by the end of next year.
How to build such a thing? Lee said his ideal Ethernet box would have 16-Tbit/s switching capacity and 80 100-Gbit/s Ethernet ports or 800 10-Gbit/s Ethernet ports.
No such box exists commercially, and Lee is reluctant to go build his own.
That leaves him with an unpleasant alternative. Lee drew up a diagram of what Facebook's future data center fabric -- that is, the interconnection of its switch/routers -- would look like if he had to use today's equipment and 10-Gbit/s Ethernet. Instead of the familiar criss-crossing mesh diagram, he got a solid wall of black, signifying just how many connections he'd need.
"I would say anybody in the top 25 Websites easily has this problem," he said later.
(Lee didn't say anything about how long it would take just to plug in all those fibers. Maybe that job could be created by funds from the U.S. Recovery Act.)
Lee also showed charts showing the disconnect between Facebook's wish list and the market. Facebook needed 512 10-Gbit/s Ethernet ports per chassis in 2007 and is likely to need 1,500 in 2010. No chassis offers more than 200 ports, he said.
Even though Lee is a veteran of Google and Cisco Systems Inc. (Nasdaq: CSCO), you might wonder if he's just one renegade engineer who doesn't represent the Facebook norm. Not really. It turns out Facebook has only five network engineers -- although Lee said that's a 20 percent increase from the spring of 2008 [math note: which means they had approximately 4.165 engineers at that time].
Even though 100-Gbit/s development started four years ago, Lee thinks it came too late, and that's got him worried about the next generation. He's pulling for 400-Gbit/s Ethernet discussions to start right away.
"Let's start the work that doesn't require money, now," he said. "If we have the standard, we can build the product later. I don't mind using an old standard."
Of course, money is a major obstacle to the next wave of Ethernet.
During an open commenting and Q&A session, multiple audience members pointed out that optical components margins are too thin to support advanced research at many companies, and that carriers are seeing their big, expensive networks getting used to make money for over-the-top services. "There's no revenue in all that bandwidth increase," one audience member commented, citing the carrier case in particular.
Because they already uses Ethernet and because they are using IP. IPoIB cannot get more than ~12 Gb/s maxing out the CPU on beefy machines. Ethernet/IB gateways are not efficient either. That's why IB is not in the datacenter: Ethernet is already there and IP traffic is important. Single vendors and debugging expertise are also problems with IB.
1) is indeed photo posting on the day after Halloween, mostly from parents, although I'm sure plenty of regrettable college photos go up too.
2) We'll give you credit even though Groundhog day ain't Feb.1st. :) It was the Super Bowl -- not as huge a bandwidth boost, maybe, but a frenzy of status updates about the game itself. That it was a really good super bowl made the posting volume really erratic. Things exploded every time something like a 100-yard interception return happened.
3) MJ, as noted before
4) Yes, back-to-school. We talk about kids using their iPhones for everything, but Lee said the greater access to computers/logon devices at school apparently does make a difference. The curves he showed were uncanny, almost identical but with one week shifted higher than the other.
1) Day after Halloween? - everyone posting costume pictures?
2) hmm, groundhog day, but that's always Feb 1; the Superbowl was that night. People posting pictures from Superbowl parties? OH! Rehasing the commercials?
No takers on the others? C'mon, #4 is pretty easy. (Think: Who was Facebook originally made for?) #1 is subtle, but ... it might help if I told you it happens on Nov. 1 every year.
I wasn't able to guess #2. It's brutally obvious, but the Feb. 1 date threw me off. It's not something I normally associate with "February."
---------------------------
1. Nov. 1, 2008, most of the morning.
2. Feb. 1, 2009, sporadic spikes throughout the evening (Eastern time)
3. June 25, 2009, one sudden, enormous burst, trailing off hyperbolically
4. Entire week of Aug. 31, 2009 -- traffic consistently higher than the previous week, for 24 hours every day (but retaining the same shape, just shifted upwards)
Sounds like a large investment for a potentially small market. But then, that is what people were saying about the CRS-1 before it sold in the Nx100's. If you are the only option, it could be a lucrative market.
It would seem more efficient to actually come up with a cost-effective and simple way to solve the meshing of multiple smaller devices to look like one large one. That way, you can invest in the development of a device that has a much larger target market. I have yet to see anyone pull this off however...
Was there any discussion of potential technologies to do this?
abashford -- He'd need 16 of them, if I undertand correctly. And more in the future, of course -- but not thousands of them, admittedly.
During the Q&A, someone actually brought up your point about volumes, and that's when Lee made the "top 25 web sites" remark, IIRC. He believes there's a hungry high-end market.
(Google apparently made similar comments earlier in the day, but I wasn't present for that session. People wanted to bring him into the Facebook discussion, but he'd already left.)
"No such box exists commercially, and Lee is reluctant to go build his own."
They could get a lot more interest from manufacturers if they could indicate to the market that more than 2 of these 'boxes' are required. :)
I have no knowledge of Facebook's architecture, does anyone know how many major data centres they maintain? Is it 2, or are they more distributed? Thanks.
The blogs and comments are the opinions only of the writers and do not reflect the views of Light Reading. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
To save this item to your list of favorite Light Reading content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Digg
Del.icio.us
Reddit
