Light Reading

The 400Gbit/s Demonstrator

Eduard Beier

A collaboration between research institutions and industrial partners demonstrated at the ISC'13 that 400Gbit/s bandwidth granularity is not only feasible, but already useful today.

For demonstration purposes a number of innovative technologies such as 400Gbit/s DWDM Super Channel, high-speed flash memory and a distributed parallel file system are used in combination.

The project is set up to have real data and realistic applications instead of yet another "hero experiment" with test generators and analyzers. Initial performance tests were performed to check the operational condition of all components working together in the demonstrator. Synthetic data and load was put on the connectivity and IT components and showed overall good operational conditions.

Then two applications were activated on the demonstrator:

  • Climate Research with centralized computing accessing to distributed data

  • Turbine Development with multi-stage processing dynamically both shifting big-data between Munich and Dresden (620km standard fiber) over a single 400Gbit/s wavelength Super Channel.

    From the research community are involved: the German Centre for Aerospace (DLR); the Leibniz Supercomputing Centre (LRZ); the Max Planck Institute for Meteorology (MPI-M) from the Max Planck Society and the computing center of the Max Planck Society in Garching (RZG); the computer center of the Technical University of Freiberg (RZ TUBAF); and the Center for Information Services and High Performance Computing Dresden University of Technology (ZIH).

    Industrial partners are: Alcatel-Lucent (NYSE: ALU), Barracuda Networks Inc. , Bull SA , Cluster Vision, EMC2, IBM Corp. (NYSE: IBM), Mellanox Technologies Ltd. (Nasdaq: MLNX), Deutsche Telekom AG (NYSE: DT), and T-Systems International GmbH .

    The fundamental network structure of the Demonstrator is shown by the following picture:

    Network Structure

    The compute clusters on both sides are clients of the distributed parallel file system which consists of 12 servers with three flash memory boards in each server. Both clusters are able to read and write at 400 Gbit/s on that file system.

    Standard Sandy Bridge servers with at least five PCIe3.0x8 slots and 128GB DRAM are used. The design goal was to create about 18 GBytes of sustained duplex data rate per node (Ethernet: 5GBytes, Memory: 6 GBytes, Fabric: 7 GBytes). The Fabric data rate has not yet been tested, the other rates have been confirmed by tests.

    For performance margin reasons the setup is moderately overbooked between memory and network. Below are the total theoretical performance numbers.

    Theoretical Throughput

    The predecessor project in 2010 (100Gbit/s Testbed Dresden Freiberg) required arrays of about 800 spinning disks on each side to form a 100Gbit/s data path. Because that setup was, for scaling reasons, not possible for 400 Gbit/s, the 2 GBytes I/O throughput flash memory board is one of the enablers of this 400Gbit/s project.

    For the ISC configuration (in the picture below), the following systems were added: a 10GbE Cluster in Dresden; a 2x100GbE link to Freiberg (not connected during ISC); in Garching the SuperMUC at LRZ (one of the fastest supercomputers in Europe) and a 480 GBytes cluster (> 2500 cores) at RZG; and a commercial Cloud Service from T-Systems (not connected during ISC).

    The Big Picture

    The project links to SDN and NFV -- in particular as it is very active in NSI (Network-to-Service-Interface) definition. (For further information, see the link below to additional project information).

    Because the "distributed high speed GPFS" approach is of some universal nature (e.g. for HPC datacenter backup and HPC workload distribution reasons) the setup will be tested for commercial applicability during the post ISC phase. The possibility to use network functions like encryption, firewalling and data compression is definitely a must in a commercial case.

    Network appliances for 40 Gbit/s and 100 Gbit/s are neither available nor would they be affordable in many cases. Therefore we are going to test virtualized network functions on standard server hardware (see picture below); that additional "module," which is based on the same server hardware as the other servers, gets in between each server and the router on each side.

    NFV Module

    Please feel free to ask questions about any technical aspect of the project. The project team, which consists of some of the best specialists on their individual field, will be happy to answer. For additional project information:

    The Partners

    — Eduard Beier, process systems engineer, T-Systems International

    (6)  | 
    Comment  | 
    Print  | 
  • Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
    User Rank: Light Beer
    9/26/2013 | 10:11:00 AM
    Re: SDN and NFV
    Ray, we are at a very early stage, but abstracting from both hardware (and in consequence IOS) layer and utilizing a common framework (see NSI WG in OGF) and at the same time allow for docking of virtalized network functions will work for multi vendor, multi carrier and multi domain. Still, its a long way to go!
    User Rank: Blogger
    9/26/2013 | 7:51:13 AM
    Re: SDN and NFV

    You say it is not restricted to a single domain, but is it applicable in networks that traverse multiple infrastructures run and managed by multiple network operators?
    User Rank: Light Beer
    9/25/2013 | 8:43:49 AM
    Re: Is this another route in for NFV?
    Requirements in industry will not be so different from those in R&D. Looking at the plans in the context of Horizon 2020 Puplic Private Partnership is a target of European efforts. Taking into account the cost of infrastructure and the prevailing attitude to pay as little as possible for utilization of infrastructure, more sophisticated multiplexing methods, and this include NFV, are inevitable.

    For network operators this scheme means significantly reduced time to market.
    User Rank: Light Beer
    9/25/2013 | 8:36:55 AM
    SDN and NFV
    In this project we will be going a step further. Our plan is to create an environment that allows for creating a virtual network based on the requirements of either applications or  carrier provided network functionality. We will not be restricted to a single domain and connectivity, our target is a network created from building blocks out of a repository. These building blocks can be connectivity with certain quality parameters or virtualized network functions like e.g. firwall functionality, compression, encryption, accelleration.


    User Rank: Blogger
    8/21/2013 | 3:00:47 PM
    Re: Is this another route in for NFV?
    NFV simply scales better than HW based approaches (if NFV tests in September show good results).

    A possible 2013 roadmap for the project:
    • fully SDN controlled network
    • 200GBit/s datapath
    • NFV in the data path 

    a possible 2014 roadmap for the project:
    • scale up to a 1TBit/s data path

    how else could you do that?





    User Rank: Blogger
    8/21/2013 | 1:01:52 PM
    Is this another route in for NFV?
    Interesting that functions virtualization takes the place of appliances that are either too expensive to deploy or have not yet been created... is NFV going to help the R&D sector more than production network operatins in the early years?
    Educational Resources
    sponsor supplied content
    Educational Resources Archive
    More Blogs from Column
    Cord cutting is a real and present threat to pay-TV, but it has been an erosion, not a landslide.
    They go together hand-in-hand in many networks but the relationship is strained – is there a way to improve the efficiency of an OpenStack and OVS deployment?
    The comms industry is rallying to the cause of open, independent interoperability testing.
    Data analytics and revenue assurance are growing in strategic importance for the cable industry.
    The introduction of IoT applications is supposed to make life easier for homo sapiens. But things are off to a bad start, according to Accedian's Scott Sumner.
    From The Founder
    The comms industry is rallying to the cause of open, independent interoperability testing.
    Flash Poll
    Live Streaming Video
    CLOUD / MANAGED SERVICES: Prepping Ethernet for the Cloud
    Moderator: Ray LeMaistre Panelists: Jeremy Bye, Leonard Sheahan
    Telecom Innovators Video Showcase
    Tail-f, Cisco & What the Future Holds

    10|9|15   |   8:17   |   (0) comments

    Steve Saunders meets with Tail-f's Director of Technology, Carl Moberg, in Stockholm to discuss becoming part of Cisco, ETSI MANO, virtualization and the need to combine science and business in order to create opportunities for service providers.
    LRTV Interviews
    Broadband Forum Embraces SDN & NFV

    10|9|15   |   02:42   |   (0) comments

    At Gigabit Europe 2015, Robin Mersh and Kevin Foster from the Broadband Forum explain how the industry body is adapting to meet the SDN, NFV and cloud needs of the access network sector.
    LRTV Interviews
    Top Tips for FTTH Operators

    10|8|15   |   02:26   |   (0) comments

    At Gigabit Europe 2015, Ventura Team co-founder Richard Jones talks about some of the key business case considerations for FTTH network operators.
    LRTV Interviews
    M-net Calls for FTTx Unity

    10|8|15   |   03:45   |   (0) comments

    At the Gigabit Europe event, Jörn Schoof from M-net, the Munich city network operator, calls for industry collaboration on fiber broadband access rollouts.
    LRTV Documentaries
    The Business Case Challenge for NFV

    10|7|15   |   03:47   |   (0) comments

    Virtual CPE is one of the early success stories for network functions virtualization, as service providers are finding flexible, programmable CPE solves a lot of logistics problems and reduces their cost. But even here, Masergy Communications faced a business case challenge, says CTO Tim Naramore.
    LRTV Interviews
    JT Offers Some Gigabit Lessons

    10|7|15   |   4:08   |   (1) comment

    Barna Kutvolgyi, managing director, Global Consumer, at JT, the incumbent operator on the island of Jersey, talks about how other service providers can learn from his company's gigabit broadband rollout experiences.
    LRTV Interviews
    AT&T's Chiosi on the Potential of Open Source

    10|6|15   |   06:27   |   (0) comments

    AT&T Distinguished Network Architect Margaret T. Chiosi talks to Light Reading's Carol Wilson about the potential for open source technology to liberate communications service providers.
    LRTV Interviews
    Network Security in a Gigabit World

    10|6|15   |   05:52   |   (0) comments

    Masergy's James Harrison talks about some of the network security and data center issues network operators need to consider as they expand their broadband services portfolios.
    LRTV Documentaries
    Telefónica: In Search of Virtual Simplicity

    10|5|15   |   07:30   |   (0) comments

    Francisco-Javier Ramon Salguero, head of Telefónica's NFV initiative, admits virtualization initially means greater complexity, but with the right abstraction layer, it is possible to create a services-driven network architecture. He explains how Telefónica's current trials and initiatives are aimed at doing that, and what his company and other carriers need to ...
    LRTV Interviews
    Gigabit Europe Takeaways

    10|5|15   |   03:47   |   (0) comments

    Participants from the inaugural Gigabit Europe event in Munich share their key takeaways from the conference.
    Women in Comms Introduction Videos
    Intel Urges Women to Take Advantage of Their Seat at the Table

    10|5|15   |   4:27   |   (1) comment

    Have inclusive and constructive conversations, attach a bigger meaning to your work and get involved in the cause, Intel's Monique Hayward advises women in comms.
    LRTV Interviews
    BT Updates on Plans

    10|2|15   |   03:16   |   (2) comments

    Peter Bell, CIO at Openreach, the access network division at UK incumbent BT, provides an update on the operator's trials and how Openreach is planning to deploy the broadband technology in its street cabinets.
    Upcoming Live Events
    October 14-15, 2015, New Orleans Ernest N. Morial Convention Center, New Orleans, LA
    November 5, 2015, Hilton Santa Clara, Santa Clara, CA
    November 17, 2015, Santa Clara, California
    December 1, 2015, The Westin Times Square, New York City
    December 2, 2015, The Westin Times Square, New York City
    All Upcoming Live Events
    Network appliances have a strong value proposition in today's networks and will continue to do so in the NFV and SDN-enabled networks of tomorrow.
    Hot Topics
    M&A Speculation Swirls Around Juniper
    Ray Le Maistre, Editor-in-chief, 10/6/2015
    Cisco's Chambers Rules Out Political Bid
    Mitch Wagner, West Coast Bureau Chief, Light Reading, 10/6/2015
    Cord Cutting? 'Fraid so.
    Brett Sappington, 10/7/2015
    Infinera Fleshes Out Its Metro 100G Story
    Ray Le Maistre, Editor-in-chief, 10/7/2015
    Cisco Makes 'Martian' Connection
    Mitch Wagner, West Coast Bureau Chief, Light Reading, 10/9/2015
    Like Us on Facebook
    Twitter Feed
    Webinar Archive
    BETWEEN THE CEOs - Executive Interviews
    With so many new and exciting communications technologies now under development, it's easy to get caught up in the industry's escalating hype cycle. That's why the ...
    Last week saw a big day in the 15-year history of Light Reading when Editor-in-Chief Ray Le Maistre and I were invited to interview the Deputy Chairman and Rotating ...
    Cats with Phones
    "What?! I'm on with Finisar about their stock price tanking" Click Here