A status report on the latest developments * What it is * What it’s good for * Who’s doing what

March 18, 2003

20 Min Read
10-Gigabit Ethernet

Now that second-generation 10-Gigabit Ethernet products are starting to arrive, the time has come to think about serious deployment of the technology.

In simple terms, it means that kinks in first-generation products will have been ironed out. It also means that vendors are now focused on ramping up production by driving down costs, just as they have been with each previous tenfold increase in Ethernet speeds.

Now service providers need to get up to speed themselves – with what 10-Gig Ethernet technology actually is, what products currently exist, and what applications it addresses.

That’s what this report is all about. Here’s a hyperlinked summary:

  • Applications

    Where 10-Gig Ethernet is likely to be deployed first

  • Market Overview

    The 10-Gig Ethernet switch market could grow at more than 40% a year

  • Technology

    Details of standards, differences between Gig and 10-Gig Ethernet

  • Components

    Devices defined, 10-Gig Ethernet transponder MSAs compared

  • Selected Systems

    Examples of leading 10-Gig products, together with key parameters

A preview of this report was given in a recent Light Reading Webinar.

Previous Light Reading reports related to this subject may be of interest as well: Metro Ethernet and 10-Gig Ethernet Transponders.— Simon Stanley is founder and principal consultant of Earlswood Marketing Ltd. He is also the author of several other Light Reading reports on communications chips, including Security Processors, Packet Switch Chips, Traffic Manager Chips, 10-Gig Ethernet Transponders, Network Processors, and Next-Gen Sonet Silicon.

Ten-Gigabit Ethernet has been developed to support a wide range of applications – from the enterprise network, through the edge and metro, into the wide area.

The Enterprise: Data Centers and Backbones

29823.gifThe figure above shows 10-Gigabit Ethernet in an enterprise network. Ten-Gig Ethernet is used for the enterprise backbone, both within a campus and between campuses, and to connect the server farm in the corporate data center.

The hottest market today for 10-Gigabit Ethernet is in the data center. The performance of servers has been increasing significantly with clock rates up to 3 GHz and faster storage technology. To move data in and out of these high-performance servers, Gigabit Ethernet network interface cards (NICs) are being fitted as standard. By moving to these high-performance servers and connecting them together with 10-Gigabit Ethernet, companies can consolidate their file servers into a small number of high-capacity data centers. This consolidation can yield significant cost savings and higher system throughput. The introduction of a 10-Gigabit Ethernet NIC from Intel Corp. (Nasdaq: INTC) will further extend the benefits of 10-Gig Ethernet in this application.

The biggest market in the future for 10-Gigabit Ethernet is likely to be in the corporate backbone. The cost of Gigabit Ethernet has dropped significantly over the last 12 months, and most 10/100-Ethernet workgroup switches now have Gigabit uplinks. High-end PCs are now becoming available with integrated Gigabit Ethernet connectivity. The range of 5 kilometers limited the use of Gigabit Ethernet between campuses. With 10-Gig Ethernet transponders capable of up to 40 km, the range is no longer an issue for Ethernet and this market.

The Metro: Lighting Up the Fiber

There is a major shift in the access market away from TDM towards 10/100 Ethernet, and even Gigabit Ethernet, for business connectivity. This is opening up new demand for Ethernet in the metro. The main deployment so far has been in the Asia/Pacific region; however it is likely that this will grow in the U.S. and Europe during next year.

Using existing dark fiber, service providers can extend their Gigabit Ethernet networks into the metro edge. As demand grows, the bandwidth through a single fiber is restricted to one gigabit, unless expensive DWDM equipment is used to multiplex multiple gigabit feeds. By using 10-Gigabit Ethernet, this multigigabit bandwidth can be achieved at significantly lower cost.

In the metro area, 10-Gigabit Ethernet can be deployed in either a star or ring topology. Unlike Resilient Packet Ring (RPR), these networks use the standard Ethernet MAC protocol. The latest 10-Gigabit Ethernet metro switches can provide network reliability similar to those based on Sonet/SDH rings.

“At Foundry Networks we support the Metro Ring Protocol (MRP), which is designed to work with standards based on 10-Gigabit Ethernet and Gigabit Ethernet and offers sub-second protection,” says Chandra Kopparapu, director of product marketing at Foundry Networks Inc. (Nasdaq: FDRY).

The WAN: Limited Deployment... So Far

Ten-Gigabit Ethernet has been designed to support the wide-area network with WAN-specific physical layers. The combination of a very significant installed base of Sonet/SDH and the lack of major infrastructure investment by carriers since its introduction has limited the deployment of 10-Gig Ethernet in the WAN.

One important application for 10-Gigabit Ethernet in the WAN is grid computing – the subject of a Light Reading Webinar in February 2003. Huge server farms spread across multiple locations across the wide area are interconnected using Gigabit and 10-Gigabit Ethernet to form massive virtual supercomputers. One example of grid computing is the TeraGrid project launched by the National Science Foundation (NSF) in August 2001.

Storage: Fibre Channel or iSCSI

The storage market is seen as a likely killer application for 10-Gigabit Ethernet. The ratification of the iSCSI standard last month by the Internet Engineering Task Force (IETF) is key to supporting storage services over IP and 10-Gigabit Ethernet (see iSCSI Gets Go-Ahead). Most storage networks today are based on dedicated storage technologies such as Fibre Channel, though there are already significant deployments of 10-Gig Ethernet in the data center.

“We see demand for 10-Gigabit Ethernet in storage applications that are based on IP, not necessarily iSCSI at the moment,” notes Foundry’s Kopparapu.

For the next-generation 10-Gigabit storage network there is now a choice. On the one hand, there is 10-Gigabit Fibre Channel; on the other, there is iSCSI running over 10-Gigabit Ethernet. Fibre Channel is a well understood technology for this application. However, 10-Gig Ethernet and iSCSI could bring significant cost savings – both through less costly equipment and the integration of storage and corporate networks.

“People are hanging on to their SANS right now. It is technology they are comfortable with,” says Rob Quiros, director of product marketing at Force10 Networks Inc. “People are essentially running two parallel networks. They are running their server farms with Gig Ethernet or 10-Gigabit Ethernet connecting out to the LAN or the metro area; and they are using SAN and Fiber Channel on the back end to connect to the storage. Ten-Gigabit Ethernet gives them the ability to collapse these two networks into one. That is when the performance is going to be crucial.”

The first 10-Gigabit Ethernet equipment was announced at the end of 2001. In 2002, more than 22 vendors took part in interoperability demonstrations at Networld+Interop in Las Vegas and at Supercomm in Atlanta (see Vendors Show Off 10-GigE at N+I and 10-GigE Vendors Get Cold Feet).

According to In-Stat/MDR, 10-Gig Ethernet only shipped about 500 ports in the first half of 2002. This initial equipment deployment was primarily to educational and research establishments. Now that 10-Gigabit Ethernet has been proven in the field, more extensive rollouts are underway.

“If you lease a dark fiber, you can simply light it up with 10-Gigabit Ethernet,” says Foundry’s Kopparapu, ”and then connect your campuses in a ring or star or whatever topology you choose.”

In-Stat/MDR forecasts core LAN switch revenues to reach nearly $25 billion by 2006, with a 43 percent CAGR (compound annual growth rate). With port costs continuing to come down, In-Stat/MDR is forecasting a significant increase in the number of Ethernet ports – especially for Gigabit and, to a lesser extent, 10-Gigabit Ethernet.

This growth in gigabit connectivity within the enterprise is building a huge latent demand for 10-Gig Ethernet in the backbone. The key to turning this latent demand into system shipments is the reduction in 10-Gig Ethernet systems cost that is now just starting to hit the street, as companies introduce second-generation products.

Ten-Gigabit Ethernet already has significant cost advantages over competing technologies such as OC192 (10 Gbit/s) Sonet and a one-year advantage over 10-Gigabit Fibre Channel for storage networks. The cost of a 10-Gigabit Ethernet port today can be as low as $17,000 – or $35,000 with the chassis included – making this relatively cost-effective bandwidth.

Switches and routers with 10-Gigabit line cards are available from a large number of companies including Cisco Systems Inc. (Nasdaq: CSCO), Extreme Networks Inc. (Nasdaq: EXTR), Force10, and Foundry. Intel is the first company to introduce a 10-Gigabit Ethernet NIC. (Products from these companies are examined in Part 6.)

The IEEE802.3ae 10-Gigabit Ethernet has been developed from earlier Ethernet standards.

The original Ethernet standards worked over a shared medium, i.e., a single cable connecting all the nodes together. Starting with 10Base-T, the network changed from a single cable to a star topology with a repeater at the center. Although there were now two wires between each node and the repeater, the network was still a shared medium and the total network bandwidth was only 10 Mbit/s. By replacing the repeater with a switch, 10Base-T can support full duplex traffic – 10 Mbit/s in each direction between each node and the central switch. The total network bandwidth is now the sum of the links to and from the switch, although the bandwidth may be limited by the switch’s internal architecture. With 100-Mbit/s Ethernet and Gigabit Ethernet, the bandwidth on each port of a switch is set following auto-negotiation between the node and switch. Some ports support 10-Mbit/s, 100-Mbit/s or 1-Gbit/s Ethernet.

A key part of the Ethernet protocol has been the CSMA/CD collision detection mechanism used to arbitrate for the shared media. Ten-Gigabit Ethernet is the first Ethernet technology to become entirely full duplex and therefore does not support CSMA/CD.

To accelerate time-to-market for Gigabit Ethernet the Gigabit Fibre Channel PMD (Physical Media-Dependent) layer was used for the optical interfaces. The 10-Gig Ethernet standard has now leapfrogged over Fibre Channel development, and therefore several new optical PMDs have been developed. Now the 10-Gig Fibre Channel standard is expected to use a 10-Gigabit Ethernet PMD.

Gigabit Ethernet uses the 8B/10B coding to include additional symbols at the PMD layer. Unfortunately, this increases the signal frequency to 12.5 GHz. To achieve the 10-Gbit/s bandwidth and limit the signal frequencies as close to 10 GHz as possible, a new, more efficient 64B/66B coding was developed. Ten-Gigabit was originally defined as an optical-only network, but there is now work in the Institute of Electrical and Electronics Engineers Inc. (IEEE) to support copper media (see Startups Move 10-Gig to Copper).

A significant goal for 10-Gigabit Ethernet has been to stretch the range from the 5 kilometers in Gigabit Ethernet to 10 and 40 km. There are now a number of companies developing proprietary optics to take Ethernet to 80 km, 100 km, and beyond.

There has been some deployment of Gigabit Ethernet in the WAN. A significant hurdle, however, has been the different data rates supported by Gigabit Ethernet and 622-Mbit/s or 2.5-Gbit/s Sonet/SDH. Unlike all other Ethernet standards, 10-Gig Ethernet is close to a Sonet/SDH data rate (OC192/STM64). To improve compatibility further, the IEEE has defined the WAN PHY, which will interface directly with 10-Gbit/s Sonet/SDH networks. The WAN PHY includes a WAN interface sublayer (WIS), which is a cut-down Sonet framer.

Table 1 shows the main differences between 10-Gigabit Ethernet and Gigabit Ethernet.

Table 1: Gigabit vs 10-Gigabit Ethernet

Gigabit Ethernet

10-Gigabit Ethernet

CSMA/CD + full duplex

Full duplex only

Leveraged Fibre Channel PMDs

New optical PMDs

Reused 8B/10B coding

New coding schemes 64B/66B

Optical/copper media

Optical media only

(copper in development)

Support LAN to 5 km

Support LAN to 40 km

Carrier extension

Throttle MAC speed for WAN

-�� Use Sonet/SDH as Layer 1 transport



Ten-Gigabit Ethernet has the same layers as other Ethernet technologies: Media Access Control (MAC), Physical (PHY), and Physical Media-Dependent (PMD). Ten-Gig Ethernet also supports all the existing Ethernet technologies, such as group trunking, and is therefore simple to introduce into an existing Ethernet-based network.

Interfaces between the layers have been standardized. The 10-Gig Ethernet MAC is connected to the PHY through either the XGMII interface or the low pin-count alternative, XAUI. The XAUI interface has 4x 3.125-Gbit/s channels with 8B/10B encoding. XAUI can also be used to connect the PHY to the PMD.

The IEEE has defined a total of seven optical port types for 10-Gig Ethernet; these are shown in Table 2, together with two proposed standards, for copper cable, shown in italics.

Table 2: IEEE 802.3ae Port Types

Device

Application

Range

Optics

Cable

10GBase-LX4

LAN

300m MMF/

1310nm

Multimode or

10km SMF

WWDM

singlemode

10GBase-SR

LAN

300 m

850nm

Multimode

10GBase-LR

LAN

10 km

1310nm

Singlemode

10GBase-ER

LAN

40 km

1550nm

Singlemode

10GBase-SW

WAN

300 m

850nm

Multimode

10GBase-LW

WAN

10 km

1310nm

Singlemode

10GBase-EW

WAN

40 km

1550nm

Singlemode

10GBase-CX4

LAN

15m

-

4 x Twinax

10GBase-T

LAN

25-100m

-

Twisted pair



The 10GBase-LX4 interface has four low-cost lasers and supports both multimode (MMF) and singlemode (SMF) fiber. There are three versions of both the LAN and WAN interfaces supporting multimode fiber for short distances and singlemode fiber for longer distances. The 10-Gigabit Ethernet distances are defined as 300 meters for short reach (SR), 10 km for long reach (LR), and 40 km for extended reach (ER).

“Most enterprises have pulled mixed cable plant,” says Force10’s Quiros. “So they have both multimode and singlemode fiber in their backbone. That has given them a lot of flexibility in the way they can deploy 10-Gigabit Ethernet.”

The majority of 10-Gig Ethernet deployment so far has been in the campus backbone and between campuses. The 10Gbase-LR and 10GBase-ER are the most suitable interfaces for these applications. Pre-standard 10-Gig systems from Cisco featured a proprietary interface called 10GBase-EX, which supports a 50km range.

Within the IEEE there is now work to develop two copper interfaces. The first, 10GBase-CX4, uses existing InfiniBand IB4X connectors and four Twinax cables. The second, 10GBase-T, is a totally new development to support 10-Gigabit Ethernet over 25 to 100 meters of standard twisted-pair cable.

10GBase-CX4 borrows much from Infiniband and is designed for short reach, data center applications. By using existing connectors and cables, the standards group expects to achieve full ratification by the end of 2003.

During 2001 semiconductor companies started rolling out 10-Gigabit Ethernet products. Devices available now include:

  • Discrete media access controllers (MAC)

  • Physical layer devices (PHY)

  • Sonet/SDH framer/mapper devices

  • Gigabit Ethernet switch devices with integrated 10-Gig Ethernet MACs

We are now seeing the introduction of second-generation silicon products with lower cost and power together with higher integration. A major focus for semiconductor companies during 2003 will be the introduction of devices to support the next generation of optical transponder and the integration of MAC and PHY in a single device.

The first 10-Gigabit Ethernet systems used either proprietary optical modules or transponders conforming to the 300-pin MSA (multisource agreement). These are relatively large and expensive modules initially developed to support Sonet/SDH.

The recent introduction of Xenpak transponders is a significant step towards expanding the 10-Gig Ethernet market. There are seven companies now shipping, or planning to ship these transponders:

Xenpak transponders are significantly smaller and cheaper, enabling more cost-effective solutions with up to four 10-Gbit/s ports per line card. They are hot-pluggable, allowing a “pay as you go” approach, with the expensive transponder modules being added as additional ports are required.

The first systems with Xenpak transponders are now entering the market. Future cost reductions will be driven by the introduction of even smaller transponders based on the XPAK, X2, and XFP MSAs.

XPAK and X2 transponders use the same XAUI interface as Xenpak, but will also support 10-Gigabit Fiber Channel. XPAK and X2 transponders are 40 percent smaller than Xenpak transponders. The XPAK and X2 MSAs are very similar, and it is unclear at the moment which will be more widely used (see The X-Wars: Agilent Strikes First).

XFP transceivers, that integrate a new serial interface (XFI), will further reduce system cost. They have a very small footprint and will support 10-Gigabit Fiber Channel and 10-Gbit/s Sonet/SDH as well as 10-Gigabit Ethernet (see XFP Module Group Debuts Spec).

This section will look at several products from a few of the many companies now delivering 10-Gigabit Ethernet enabled systems. These include core routers, enterprise and metro switches, and an NIC. These products are summarized in Table 3: Table 3: Typical Systems

Company

Chassis

Max Number of Blades per Chassis

Max I/O Bandwidth

Max Switching Bandwidth

10GE Blade

10GE Ports per Blade

Optical Module

10GE Port Types

Switch Bandwidth per 10GE port

10GE Ports per Chassis

Cisco

12400 Series

15

150Gbit/s

160Gbit/s

10GE 10km

1

Proprietary

10GBase-LR

10Gbit/s

15

10GE 40km

1

Proprietary

10GBase-ER

10Gbit/s

15

Catalyst

12

120Gbit/s

128Gbit/s

10GE 10km

1

Proprietary

10GBase-LR or 10GBase-ER

10Gbit/s

11

Extreme

Black Diamond

16

160Gbit/s

128Gbit/s

10GLRi

1

300 pin

10GBase-LR

8 Gbit/s

16

Force10 Networks

E Series

14

280Gbit/s

640Gbit/s

2-port LAN 1310

2

300 pin

10GBase-LR

40 Gbit/s

28

2-port LAN 1550

2

300 pin

10GBase-ER

40 Gbit/s

28

2-port WAN 1310

2

300 pin

10GBase-LW

40 Gbit/s

28

Foundry

BigIron NetIron FastIron

15

300Gbit/s

120Gbit/s

LAN 850

1

300 pin

10GBase-SR

8 Gbit/s

15

LAN 1310

1

300 pin

10GBase-LR

8 Gbit/s

15

LAN 1550

1

300 pin

10GBase-ER

8 Gbit/s

15

2 Port XENPAK

2

XENPAK

10GBase-LR or 10GBase-ER

4 Gbit/s

30

Intel

PRO/10GbE LR Server Adapter

1

10Gbit/s

-

PCI-X Card

1

XENPAK

10GBase-LR

4 Gbit/s

1



In most cases there are a range of systems available, typically with four, eight, or 16 I/O slots.

For each family, the maximum number of I/O slots is shown. This capacity is assuming no redundancy. Some systems, such as the Cisco 12400, use one of the slots to support a 1:1 redundant switch fabric, reducing the number of I/O slots, in this case, to 14. Other systems, such as the Extreme Networks Black Diamond, support graceful degradation of performance if a switch card fails but do not support 1:1 redundancy.

The fourth column shows the maximum I/O bandwidth with 10-Gigabit Ethernet blades fitted in each slot. For systems, like the Foundry Networks BigIron family, that can support either a single or dual 10-Gig Ethernet blade, it is assumed that dual 10-Gig Ethernet blades are fitted in all available slots. All of the products listed here, except the Intel Server card, can support Gigabit Ethernet ports as well, and most support a range of ports including Sonet/SDH and TDM.

The fifth column shows the maximum switching capacity of a fully loaded system. Some systems, such as the Extreme Black Diamond and Foundry BigIron, only support 8-Gbit/s switching bandwidth per I/O slot. The Force10 Networks E Series is the only system with enough raw switching bandwidth to support 40-Gbit/s per I/O slot.

“We have a next-generation general purpose platform,” says Force10’s Quiros. “We have targeted the group of users that need next-generation capability and full line-rate 10-Gigabit Ethernet and Gigabit Ethernet with advanced capabilities like access lists and QOS, running simultaneously without impact on performance.”

The second half of the table shows the type of blade and the number of 10-Gig Ethernet ports per blade. Included are details of the type of optical module fitted and the 10-Gig Ethernet Port types supported. The Foundry 2 Port Xenpak is the only announced blade supporting Xenpak transponder modules. These blades, like the Cisco Catalyst blades, can be shipped with one of two types of optical module.

“Our first 10-Gigabit Ethernet products started shipping in Q4 of 2001 for revenue and was the industry's first 10-Gigabit Ethernet product to ship,” says Foundry’s Kopparapu. “We are now introducing a second-generation 10-Gigabit Ethernet module, which is based on a Xenpak optics and supports both 10GBase-LR and 10GBase-ER.”

Finally, for each blade type, the table shows the maximum, switched bandwidth available for each 10-Gig Ethernet port and the maximum number of 10-Gig Ethernet ports that can be supported in a single chassis. The Foundry 2 Port Xenpak blade has a switched bandwidth of only 4 Gbit/s – however, this matches the maximum bandwidth supported by the Intel PCI-X server card. The PCI-X bus limits the Intel server card bandwidth, and any other similar card, to 4 Gbit/s.

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like