Metro Ethernet

Ethernet has had an incredible run since the technology was invented in 1973 by Bob Metcalfe, working at Xerox Corp.'s (NYSE: XRX)famous Palo Alto Research Center (PARC) in California. It's a great example of standardization expanding markets, bringing down prices, and delivering real benefits to end users – not to mention the billions of dollars it's made companies like Cisco Systems Inc. (Nasdaq: CSCO) and Intel Corp. (Nasdaq: INTC).

Now, of course, Ethernet is everywhere inside buildings and is finding its way into telecom networks, either as a technology supporting services such as virtual private networks (VPNs) or as a service in itself, enabling businesses to link together Ethernet LANs in different locations.

The frontier is the metro network. This is where users' Ethernet traffic hits traditional telecom transport technologies, notably Sonet or its non-North American equivalent, SDH (Synchronous Digital Hierarchy).

Bringing together these two technologies is a complicated business – partly because Ethernet wasn't designed for wide-area networks and partly because there are all sorts of ways of making it more carrier friendly. The potential market, like all Ethernet markets, is huge, so just about every vendor of telecom equipment is promoting a solution.

The bottom line is that the metro Ethernet market is a battlefield. Confusion reigns, and no clear winners have yet emerged in terms of particular approaches to technical issues, or in terms of equipment vendors or service providers having won an upper hand.

All the same, it's important to understand what's going on, because this is where the future of telecom will be decided. That's why we've prepared this exhaustive report on the subject, as part of our series on metro technologies.

The first metro report – Metro Multiservices Evolution – gave a general overview and explained how different technologies are likely to be combined in metro networks. The second report – Next-Gen Sonet – dug into the details of how Sonet/SDH is being extended to handle packet-based traffic in a better way. This report covers Ethernet, and a further report will cover Resilient Packet Ring (RPR) technology. After that, we'll be delving into the optics with a report on metro DWDM developments.

Here's a hyperlinked summary of this report:

Some of this report digs quite deeply into technological issues. To get the most out of it, why not start by listening in to the archived Web-enabled preview, given by Mary Jander, Senior Editor at Light Reading. Click here to do this.

Here's some background reading that might also help:

— Introduction by Peter Heywood, Founding Editor, Light Reading
http://www.lightreading.com

— About the author: Tim Hills is a freelance technical writer. He may be reached at: [email protected].

Want to know more? The big cheeses of the optical networking industry will be discussing this very topic at Opticon 2002, Light Reading’s annual conference, being held in San Jose, California, August 19-22. Check it out at Opticon 2002.

Register now and save $500 off the registration fee. Just use the VIP Code C2PT1LHT on your registration form, and deduct $500 from the published conference fee. It's that simple!

Ethernet has moved center stage in the metro world in the last couple of years, but the task of turning vision into reality is proving tougher than expected.

Metro Ethernet as a service still seems to have a head of steam, even though some of the startups in this field have run into turbulent times. Some of them have been shut down (see Another Metro Provider Fails: Was Vendor Financing the Difference? and WaveCrest Acquires Storm Telecom); but others have survived. Yipes Enterprises Services Inc., for instance, has risen from the ashes of Chapter 11 bankruptcy (see Yipes Reborn – Amid Accusations). And Telseon Inc. says it's close to turning a profit (see Telseon: Profitable in 2003?).

It’s less clear that metro infrastructures are going to be totally rebuilt. But carriers can’t afford to ignore Ethernet – not least because it promises to revolutionize their customers’ approach to IT.

Even in the midst of today’s telecom bust, business end-users worldwide are still spending an awful lot of money on WAN and Internet access services – $82 billion a year, according to a May 2002 study from Infonetics Research Inc., called "User Plans for WAN and Internet Access" (see Infonetics: WAN Spending Grows Steadily).

And, although growth is no longer at dotcom bubble levels, Infonetics’ estimate of a 26 percent increase to $103 billion by 2006 is not to be sneezed at in an industry where carrier bankruptcies have reached epidemic proportions.

But if connectivity demand is still in robust health, can carriers make it pay? In a world in which competition creates – and high IT usage demands – low bandwidth pricing, carriers are going to have to get a lot smarter, both in controlling their own costs and in more closely matching the networking needs of their business customers. The traditional approaches are no longer enough.

"The WAN and Internet access world is changing from one dominated by Frame Relay and TDM leased lines to one that uses a mix of Frame Relay, TDM, optical/Ethernet, and broadband technologies such as DSL, cable, and fixed wireless," says Jon Cordova, Infonetics analyst and lead author of the recent study. "Increasingly, organizations of all sizes are using multiple technology types along with Internet VPNs to build and secure their WANs."

It is the belief in the inevitability of such a change that has powered the extraordinary rise in interest over the last few years in Ethernet – along with IP – as a (or possibly the) future metro network technology. The argument has a beguiling simplicity:

Why, indeed? Especially as there are strong arguments that a carrier-class metro Ethernet could be cheaper and more flexible for carriers than conventional WAN technologies such as TDM (time division multiplex), Sonet/SDH, and ATM (asynchronous transfer mode). It would certainly be a massive plus for end users to be able to regard the metro and WAN as a super Ethernet LAN into which they could just plug their own LANs and manage them in the usual way.

Not so simple a story

It is important to be very clear about what is going on with this argument, because appearances are deceptive. Pure Ethernet enthusiasts – let’s call them Etheroids – interpret it literally. This means that not only does Ethernet appear as the end service to the customer, but that it provides the Layer 2 switching and the base bitstream transport in the metro network. So the metro not only looks like a native Ethernet in terms of what it delivers to the customer, but it is a native Ethernet in terms of how it works. Pretty radical, really, if you are used to carrier-proof things like Sonet and ATM, especially should you then intend to use such native Ethernet as the bearer for your TDM and other legacy services as well.

But there is another interpretation. This one – its supporters just have to be called Soneteers – restricts Ethernet in the metro to its Layer 2 functions only. This means that something else – usually Sonet, as there is so much of the stuff about – has to provide the Layer 1 transport. This is a more comfortable position for traditionalists, as Sonet has the same de facto status in the carrier world as Ethernet does in the business world – and it is supremely good at carrying all that bread-and-butter legacy TDM traffic that pays a lot of the bills these days.

Some reduce the Soneteer view even further by eliminating Ethernet switching from the metro and pushing it out to the customer premises. For these Minimalist Soneteers metro Ethernet means using the advanced features of next-generation Sonet to provide finely tuned and dynamic Ethernet bandwidth via simple interfaces on premises-to-premises and premises-to-POP pipes (see Next-Gen Sonet and Next-Gen Sonet Silicon). This is essentially a straightforward point-to-point Ethernet private-line service with no pretensions to high functionality – but it can be effective enough as an initial service.

Just to add a final touch of complication, Ethernet and Sonet can be combined into single hybrid system platforms. Appian Communications Inc., for example, does this with its OSAP Optical Services Activation Platform, where the underlying transport can be either Sonet or Ethernet. The intention is to support current TDM services natively via Sonet, but with the option of migrating completely to Ethernet in the future, if required.

The result of the Etheroid/Soneteer split is that it is very easy to get at cross-purposes when talking about metro or optical Ethernet (another favorite industry term). As David Cunningham, the strategic marketing manager for the networking solutions business unit of chip and device vendor Agilent Technologies Inc. (NYSE: A), observes: “I don’t think there is a common understanding in the industry of what metro Ethernet is – or optical Ethernet. Those were two things that were used during the technology bubble, but not very precisely. I think there will be some companies that try to use native Ethernet on dark fiber, but given the state of the industry, it remains to be seen just how popular that will become.”

In practice, many vendors take an intermediate position – say, under such-and-such circumstances, Ethernet transport here, but not there – so metro Ethernet has a range of interpretations, depending on whom you are talking to. Like next-gen Sonet, it is more a brand or slogan than a single technology or network architecture.

Many argue that metro Ethernet includes exploiting modern fiber-based 1- and 10-Gigabit Ethernet in the metro somewhere. But there is a wide range of views on the extent to which native Ethernet could, or should, be used for underlying transport and for traffic engineering (aggregation and grooming), protection, and so forth. And in which parts of the metro? Access (including the first mile)? Or core?

Adding a further dimension of complexity is whether Ethernet, however transported, will become a form of universal service layer, so that customers could access other services (such as voice or Frame Relay) through a ubiquitous Ethernet wall socket.

Says Karen Barton, VP of marketing at Appian Communications: “When we talk about Ethernet we really do come from the perspective of Ethernet as a service-enabling technology. Ethernet becomes the universal service interface into many different services, which could include Frame Relay, Internet access, and even a virtual private LAN. These services are independent of geography and the underlying transport, but they all build on the simplicity and scale of Ethernet. That’s the real proposition that Ethernet brings to the table.”

Not surprisingly, some of the members of the recently formed Metro Ethernet Forum (MEF) take the Etheroid pure-Ethernet view, although the MEF itself has a broader remit and involves vendors from the Sonet side of the industry.

According to Nan Chen, president of the MEF and director of product marketing at Atrica Inc., the MEF considers metro Ethernet to cover both Ethernet transport and Ethernet services to end users. “Obviously, there are multiple ways to provide Ethernet services. However, we believe that metro Ethernet is one of the best transports to deliver those services. You don’t have to do a lot of protocol translations, for example. That’s why one of the MEF’s primary priorities is to make Ethernet-based transport carrier-class. Of course, the other primary priority is to define Ethernet services for all feasible transports.”

Nevertheless, substantial sections of the industry lean towards the Soneteer side as far as the initial wide-scale deployment of metro Ethernet is concerned. They point to the huge embedded base of Sonet networks (which it is financially impossible for incumbents to write off and replace, even if they wanted to), the data-friendly next-gen Sonet technologies now reaching the market, and Sonet’s exceptional, carrier-class OAM capabilities, including fast automatic protection – capabilities that are difficult to emulate in Layer 2 Ethernet.

“Our view is that the market is going to go for Ethernet over Sonet as a starting place,” says Nathaniel Grier, director of optical networking solutions at chip and device vendor Agere Systems (NYSE: AGR). "Will it migrate to something after that? Probably. For example, it could be Ethernet over GFP [generic framing procedure] over OTN [optical transport network]; that seems to be a likely candidate. It all boils down to network administration and being able to detect errors and faults in the network, and to provision around them automatically." Many service providers, he says, are taking Ethernet-over-Sonet approaches.

But even with the preponderance of Sonet-based networks, metro service providers and their circumstances are very diverse; and many vendors, including those with a bias towards pure Ethernet, expect service providers to tend to adopt a mix-and-match approach in the longer term.

“We have no delusions about Ethernet being everywhere. Instead, we argue that in the future all carriers will use Ethernet somewhere in their network – whether it be as an access technology, in the metro core, inside the CO [central office], as a cost-effective point-to-point connection between upstream and downstream providers, and so on,” says Steve Garrison, director of corporate marketing at switch/router vendor Riverstone Networks Inc. (Nasdaq: RSTN). “With this as a driving philosophy, over the next several years we see legacy networks as still a dominant piece of every carrier’s network. It’s really the combination of next-generation Ethernet expertise with strong support for legacy interfaces that is required from metro routers as the metro evolves.”

He points out that about 30 percent of his company’s business involves the support of traditional WAN transport – T1/E1, T3/E3, Sonet/SDH, and ATM – rather than native Ethernet.

Ashok Madanahalli, service provider marketing director at Extreme Networks Inc. (Nasdaq: EXTR), expresses a similar view: “Obviously we cannot ignore Sonet. Even in the Metro Ethernet Forum – and we are a member – when people ask us about how we view Sonet/SDH we say that we believe that Sonet is a part of our focus, with product support for IETF RFC 2878 mapping of Ethernet VLANs into Sonet timeslots, as we realize there is a huge deployment of Sonet out there. For new carriers for new areas, for areas where you need to have a cost-effective solution, where you need to have a simpler solution, where you need to have new services for deployment and installation – really, Ethernet becomes very viable.”

All friends, really

Such comments show that it would wrong to overemphasize the Etheroid/ Soneteer divide and the either/or choice it implies. The metro industry has clearly been chastened by its current financial problems, and most people are becoming very pragmatic in their approaches to Ethernet. Further, there are several specific factors tending to push the two sides towards a middle ground of mixing and matching approaches.

First, there is something of a convergence effect at 10-Gbit/s between Sonet and Ethernet. Not only is this the first point at which the Sonet multiplexing hierarchy closely approaches the Ethernet family of extensions, but it is a useful size of big pipe for the metro core and an obvious candidate for a standard metro building block. Unsurprisingly, dual-purpose devices are emerging that will support either Sonet or Ethernet at this rate, lessening the pressure on vendors to make the "right" technology bet.

Says Julie Eng, director of product development for optical transceivers and transponders at Agere: “If you look at it from an optical front-end perspective, for the type of optics, laser, detector, and analogue IC, the Ethernet or Sonet specs are pretty similar. Frankly, the majority of the difference is on the mux/demux-type IC application and the protocols. We make ICs for the Ethernet and Sonet markets. So if you converge those so that the parts you introduce can achieve both Ethernet and Sonet, everybody wins, because you use the leverage of the volumes of both markets to bring down the cost for everyone.”

Second, there is a growing emphasis on Ethernet’s potential role in the access network generally, and access has always been the most diversified area in telecom as far as technology is concerned. Both 1- and 10-Gigabit Ethernet have natural initial applications for basic LAN extension and big corporate uplinks to the metro access or core networks through their longer-range LAN and WAN physical interfaces over dark fiber, or via the 10-Gigabit Ethernet WIS Sonet-compatible interface.

And the idea of a public Ethernet to provide wide-scale broadband access for business and residential users is gaining acceptance. The Ethernet in the First Mile Alliance (EFMA) was formed last year specifically to promote the Institute of Electrical and Electronics Engineers Inc. (IEEE) 802.ah standards, due around mid-2002, for Ethernet as a true universal broadband access mechanism – over fiber or copper pairs (see Extreme Launches Ethernet Alliance, Ethernet Frame Relay , and Ethernet Over Copper: Now You’re Talking).

Finally, there is the timing effect. Both OC192 and 10-Gigabit Ethernet are new technologies as far as metro cores are concerned. As carriers come under pressure to upgrade because of capacity constraints, there is potential for overlay solutions that introduce Ethernet incrementally in parallel with an existing Sonet architecture to accommodate bandwidth expansion. This can significantly enhance the economic attractiveness of Ethernet, according to Nan Chen.

“According to a recent study from Network Strategy Partners, implementing an optical Ethernet in coexistence with an existing Sonet infrastructure delivers a strong ROI within a year, as opposed to three years for upgrading Sonet,” he says. “Atrica is presenting a solution that integrates optical Ethernet and existing Sonet to build an overlay network cost-effectively. That is what the bottom line is – cost-effectively building a network for offering profitable Ethernet services.”

So what might carriers hope get out of the various approaches to metro Ethernet? Generally, they fall under two headings:

But the details depend on which form of metro Ethernet is implemented, and both are interrelated.

A metro Ethernet service can provide an improved fit with end-customer needs by eliminating the hassle and expense of meshing Frame Relay or ATM PVCs (permanent virtual circuits) in the metro or WAN to support LAN extension, for example. Instead, by creating a metro-scale Ethernet network, carriers can put customer sites onto, say, a virtual LAN and simply accept – via Bridge Control Protocol (BCP), for example – Ethernet frames from customer routers. Further, a metro Ethernet is naturally multicast (unlike Frame Relay or ATM PVCs), and customer bandwidths can be easily adjusted through small increments. So metro Ethernet is an inherently flexible service offering.

Just about everyone in the industry expects easily scaleable Ethernet, with bandwidths incremented in small steps from 1 Mbit/s to 1 Gbit/s, to be immensely attractive to end users of all types for such applications as Internet access, point-to-point private lines, and transparent LAN service. If customers can easily increase bandwidth in small, economical steps, they are very likely to do so, thereby boosting overall traffic usage. And making a wide range of economical bandwidths available will help to bring in smaller customers and expand the market for new revenue-enhancing services.

Epana Networks, a new metro Ethernet operator in New York, exemplifies these types of possibility. It is using a combination of fiber and conventional T1 and T3 accesses to offer a range of services transported by native Gigabit Ethernet to a wide range of customers, and especially to smaller businesses. Its Epana Ethernet Everywhere service includes:

There is also an emerging market in some developing countries for deploying Ethernet services to small businesses initially and perhaps longer-term to residential customers. In places like China, India, and various South American countries, where many Sonet/SDH rings would typically be OC3/STM1, a granularity of 50 Mbit/s is far too coarse – an Ethernet-based granularity of 1 or 2 Mbit/s is more realistic.

On the network side, the hope is that Ethernet, being essentially a straightforward technology, will prove generally cheaper than current Sonet-based networks in terms of equipment, interfaces, service provisioning, and general operations and maintenance. This will lead to savings for customers as well.

There is no doubt that Ethernet is inherently simpler than Sonet, and thus has a built-in cost advantage.

Ray Milhem, Extreme Networks’ senior director of product management, points out that it is simpler and cheaper to connect packet switches such as his company’s BlackDiamond 6800 Series into Ethernet ports on add/drop multiplexers, instead of, say, using packet-over-Sonet. “The [POS] optics are more expensive and the performance is not as good, due to the immense amount of manipulation you have to do. So the data-aware capabilities that people are bringing in on the add/drop side gives us the physical Ethernet port we plug into – but we still have to do all the manipulation of the packet and the shaping and the QOS at the edge.”

Further savings can come from approaches that use Ethernet over DWDM, as the combination of wavelength switching and Ethernet-transport traffic engineering can eliminate the need for Sonet crossconnects and DWDM transponders. This indicates that a crucial point for metro Ethernet cost savings is the integration of Ethernet switching and optical transmission.

“Optical integration in Optical Ethernet can give at least 50 percent savings on capex alone because it eliminates several network elements – Sonet crossconnects and DWDM transponders,” says Atrica’s Nan Chen.

Interested parties are thus making some pretty big claims for the size of the cost savings that native-mode metro Ethernet can deliver. The 10 Gigabit Ethernet Alliance (10GEA) says that 10-Gigabit Ethernet over DWDM can support dark-wavelength gigabit services at costs that are less than those of a conventional T3 or OC3 circuit, and the organization cites Yankee Group figures suggesting savings of around 80 percent compared to Sonet interfaces, with additional provisioning and operations savings of between 30 percent and 50 percent.

In April 2002, Atrica released a study by management consultants Network Strategy Partners to quantify the cost savings of the business use of metro Ethernet services – and the results are dramatic. The study compared the costs of three network architectures in delivering a bundle of Ethernet services (such as Internet access and transparent LAN, among others) under various scenarios. The architectures were:

The study found that the optical Ethernet scenario would give a payback in nine months, compared to 30 months for Ethernet/DWDM and 42 months for Ethernet over next-gen Sonet with DWDM. It also had a total cost of ownership 43 percent less than Ethernet/DWDM and 51 percent less than next-gen Sonet with DWDM, and it was cheaper than the other two for every element of capital and operating cost. The cost advantage stems essentially from the integration of Ethernet switching and optical transport and high scaleability.

However, this is for a greenfield implementation – and only for Ethernet-style services, too. For carriers with existing networks, which may have significant voice and other TDM traffic, the considerations become much more complex. Also, other issues than pure technology choices arise.

Says Appian Communications’ Karen Barton: “We have observed that the more carrier-class an infrastructure becomes, the less different the technologies appear to be at the cost level. If you are building a service where you are guaranteeing 99.999 percent availability, you will have to have a full backup strategy, including the bandwidth necessary to support premium services, when there are failures in the network. That adds cost to the network.”

So the rather boring implication is that the cost savings of metro Ethernet will depend greatly on how you implement it, which in turn depends on what you already have and what your aims are. For established carriers, the greenfield cost models are probably best regarded as a source of inspiration.

However, things are much more definite on the service side. The industry appears pretty clear that, in addition to dynamic bandwidth provisioning and support for ASP and storage services, there are three key early applications of metro Ethernet:

VPN capabilities and scaleability are crucial for the attractiveness of metro Ethernet but have to be looked at within a carrier’s network as a whole. Says Riverstone Networks’ Steve Garrison: “Carriers want a VPN that is scaleable, with bandwidth on demand, that can be done at Layer 2 – meaning on the MAC address – or at Layer 3 – meaning the IP address. Whether they want to attack it from a Layer 2 or Layer 3 model really depends on whether they are a telco with a lot of Frame Relay or DSL access technology, or whether they are more of an IXC/core player with a lot of BGP technology in their networks. So it’s important for vendors to be able to support both.”

The Etheroid/Soneteer contrast means that sizing up the state of metro Ethernet technology is complicated by the need to separate the underlying transport aspects from those operating at Layer 2 and above.

Both 1- and 10-Gigabit Ethernet can be run in native mode over dark fiber, with Ethernet providing the Layer 1 transport mechanism and all the Layer 2 functions.

Alternatively, one of the varieties of next-gen Sonet (covered in an earlier report in this series – see Next-Gen Sonet ) can provide Ethernet-over-Sonet capabilities to handle the Layer 1 transport and which, integrated with Multiprotocol Label Switching (MPLS) (as it increasingly is), affords some functions more commonly associated with Layer 2 Ethernet or Layer 3 IP.

And then there is the upcoming Resilient Packet Ring (RPR – to be covered in a later report in this series) technology, which can run over dark fiber or Sonet rings to provide Layer 2 switching functions.
Table 1: Selected IEEE 802 Ethernet Standards (not all fully finalized)

These capabilities, when combined, turn Ethernet into a powerful longer-range networking protocol. For example, 802.1d and 802.3x help to create a full-duplex switched Ethernet for two-way point-to-point links. There are no frame collisions (as in the original 10-Mbit/s LAN form) and theoretically no distance limitations, so, with suitable switches, you can create high-capacity low-latency networks that are highly efficient because of the small frame overheads.

802.3z 1-Gigabit Ethernet has been standardized and available from the mid-90s. The commonly used SMF 1000Base-LX physical interface (PHY) has a basic range of 5 km, but extensions are defined for up to 100 km. Vendors are still pushing performance further, however, to match metro requirements more closely.

For example, Metrobility Optical Systems released Gigabit Ethernet extenders in April 2002 that, when used in cascade, offer ranges up to 280 km. The extenders provide signal retiming, data regeneration, and wavelength conversion, but at lower costs than for straightforward repeaters (see Metrobility Stretches Gig-E and Metrobility Intros T3/E3 Interfaces).

The latest addition to the Ethernet family is 802.3ae 10-Gigabit Ethernet, which is for all practical purposes fully standardized, although the IEEE work will not be formally completed until around mid-2002. As always, the IEEE standards are concerned only with the basic physical and MAC aspects, not with higher functions that are specific to the needs of carriers. It is the need to extend Ethernet standards into the carrier space that has led to the formation of the Metro Ethernet Forum.

Ten-Gigabit Ethernet equipment is becoming available in increasing quantities. The 10 Gigabit Ethernet Alliance has organized a couple of interoperability demonstrations – one last September that attracted 18 vendors (see 10 Gig Ethernet Alliance's Big Day) and another one last May with products from no fewer than 24 vendors (see Vendors Show Off 10-GigE at N+I).

Ten-Gigabit chipsets and transceivers have also been the subject of smaller interoperability demos (see Zettacom, Xilinx Interoperate, Mitsubishi, Optillion Team on 10-Gig and Velio Participates in 10-Gig Demo, for examples).

Vendors, however, still seem reluctant to prove that their products can really deliver 10-Gbit/s speeds, judging by the recent cancellation of planned performance tests (see 10-GigE Vendors Get Cold Feet).

One reason for this might be a realization that the immediate impact of 10-Gigabit Ethernet on carrier networks may be relatively small because of the very large increase in capacity it represents.

“The rule is far more towards OC48/STM16 [2.5Gbit/s] being the bandwidth benchmark right now. It will be a while before we see a demand for 10-GigE as the transport layer technology,” says Riverstone Networks’ Steve Garrison. “Do we have the technology? Yes. Do we see revenue coming from it? No, not large amounts. We see 2003 being a stronger year for revenue in 10-GigE products.”

The IEEE Ethernet standards are also snaking nearer to the end user through the work of the 802.3ah Task Force on Ethernet in the First Mile (see IEEE OKs 1st-Mile Ethernet).

The Ethernet in the First Mile Alliance, formed in December 2001, aims to build on this effort and hopes to establish standard interface speeds and media types that will exploit 1- and 10-Gigabit Ethernet metro technology, resulting in the wide-scale deployment of native Ethernet (see Extreme Launches Ethernet Alliance and First-Mile Ethernet Building Steam).

Media considered in 802.3ah are point-to-point fiber and point-to-multipoint PONs (passive optical networks) at 1 Gbit/s over 10 km and copper twisted pair at 10 Mbit/s over 750 m. Arguably, Ethernet over PON could help reinvigorate the development of the PON as a fiber access architecture (see Ethernet in the 'Hood , Passave Grabs $7M, and Salira Prompts PON Questions).

The IEEE’s layered design for 10-Gigabit Ethernet gives considerable flexibility at the cost of a confusing number of options at the PHY Layer. The basic idea is to separate the fully duplex MAC Layer from the various PHYs by an intermediate 10-Gigabit Ethernet Media Independent Interface (XGMII) or a 10-Gigabit Ethernet Attachment Unit Interface (XAUI – pronounced Zowie in the better circles; sources are silent on XGMII). XAUI reduces the 74-pin XGMII to 16 pins, allows the interoperation of different equipment implementations, and helps to reduce costs through the use of pluggable optical transceivers, lower pin counts, and longer trace lengths for higher-density ICs and greater board-layout flexibility.

The PHYs divide among several LAN types and several WAN types (and the WAN types are available either to the standard 64B/66B line coding or to the Sonet-compatible WIS format). However, for transmission range, the WAN PHYs are equaled by several of the LAN PHYs, so it is perfectly possible to build metro connections with LAN PHYs. The basic PHYs of metro interest are:

There is also a Uni-PHY, which combines LAN and WAN PHYs in a single unit to increase product flexibility and reduce costs.

Vendors are already releasing products that exceed these specs. For example, Foundry Networks Inc. (Nasdaq: FDRY) reached 85 km with its 10GBase-ER module over OFS’s TrueWave fiber in a demonstration in May 2002 (see OFS & Foundry Demo 10-Gig).

Another crucial acronym is XENPAK, which uses the 802.3ae standard to form the 10-Gbit/s Ethernet links. It is an industry-agreed format for plug-in modules to the XAUI to support different physical media devices, and it parallels other industry multisource agreements.

“It is a very well adopted packaging method as of right now for the 10-Gig Ethernet solution,” says Shuo Zhang, Ethernet optics product marketing manager, for the fiber optics products division of Agilent Technologies. “We are widely sampling the customers now and are targeting to release the product very soon.”

Agilent’s 10-Gigabit Ethernet XENPAK optical transceiver is a LAN PHY that incorporates the physical layer functionality from the optical interface to the XAUI (four channels at 3.125 Gbit/s each) electrical interface, including both 8B/10B and 64B/66B coding and decoding. The 1310nm serial transceiver has a range of 10 km and uses Agilent’s uncooled, directly modulated laser technology.

The inexorable drive to reduce component cost, size, and power consumption is already in play in the 10-Gigabit Ethernet arena. Agere Systems, for example, recently released its uncooled TB64L1 transponder specifically for metropolitan-access applications and optimized for spans of 2 to 40 km. The use of uncooled lasers reduces the power consumption of a 40km XENPAK from approximately 9W to approximately 6W. Further power reductions come from IC-technology migration – GaAs to SiGe, SiGe to CMOS – which will reduce the power of any module, independently of the optical frontend.

Says Agere Systems’ Julie Eng: “The advantages are threefold: cost, size, and – most important – power consumption. Some of our customers on the XENPAK want to stick eight of these 10-GigE transceivers in a row on a single card. If you are going to save 3W per part, that’s [nearly] 25W, which is a significant fraction of the system power allocation. Typically, it’s a saving of between 20 percent and 25 percent [at the system chassis level]. At the module level, it’s more like 50 percent.”

Although XENPAK is proving popular, chip vendors are now proposing more compact alternatives – notably XFP, supported by a group of 44 vendors, and XPAK, promoted by Intel, Infineon Technologies AG (NYSE/Frankfurt: IFX), and Picolight Inc. (see Is Xenpak Past It?, Sizing Up Xenpak , XFP Touts Progress, and Trio Announce 10-Gig MSA).

With a lot of the technology in place, the main task for metro Ethernet is to show that the pieces can be pulled together to produce networks and services with the robustness and other characteristics that carriers expect. This is the quest for carrier-class Ethernet, and it has been exercising vendors mightily of late.

Before getting too excited about technological capabilities as defined by standards, it is as well to start with a mundane reality. As Elie Seidman, CEO of Epana Networks, points out, system stability is essential.

“Stability is the most important thing we look for,” he says. “One of the big challenges is that the switches need to be extremely stable. We are less interested in more features, and more interested in more stability. These boxes need to go years without crashing. There is tons of room for improvement around stability. That’s going to be the biggest limiting factor for widespread Ethernet deployment as a true carrier-class technology that replaces the traditional Frame Relay or ATM switches.”

The essential limitation for carriers of 1- or 10-Gigabit Ethernet over dark fiber is that it is just a best-effort technology, with none of the sophisticated protection, QOS, or OAMP (operations, administration, maintenance, and provisioning) capabilities taken as read with carrier Sonet networks.

Nan Chen says that carrier-class Ethernet is defined as embracing five key things:

“MEF is converging on the carrier-class Ethernet definition,” Chen says. “At this point the MEF is working on four things: metro Ethernet protection (50ms resiliency), overall OAM monitoring capability, QOS, and TDM over Ethernet, which is circuit-emulation services.”

By May 2002 the MEF had produced 10 draft documents, summarized in the list below. "Draft" is used more in the IEEE sense of a definite base document for further work (amendment and additions) than in the more free-floating Internet Engineering Task Force (IETF) sense, where a range of different proposals may be produced. “So the draft is a significant step towards the eventual standard – probably within the next two or three meetings, I think, we will have that ironed out,” says Chen.

Resiliency

Perhaps the biggest question mark hanging over any Ethernet-based network is resiliency. Everybody agrees that trying to reconfigure around fiber-span or node failures with the standard Ethernet Spanning Tree Protocol is a complete nonstarter for carriers, and even the Rapid Spanning Tree Protocol is far from the de facto sub-50ms Sonet benchmark.

As with many aspects of metro Ethernet, there are various proprietary vendor solutions for speeding up Spanning Tree or finding alternatives. Riverstone Networks, for example, released its Rapid Ring Spanning Tree Protocol (RRSTP) in late 2001 (see Riverstone Rings Rapidly). Riverstone claims its implementation is about 60 times faster than other forms of metro Spanning Tree technology, but it works only for Resilient Packet Rings, not for physical mesh topologies. Another proprietary ring-based protection system is Extreme Networks’ Ethernet Automatic Protection System (EAPS). This creates a primary and secondary path (East or West) for each VLAN, and switches between them when necessary; it can also allow spatial reuse.

Atrica claimed in January 2002 that its Atrica Resilient Ethernet Access (AREA) technology delivered the industry’s first sub-50ms restoration for metro Ethernet networks (see Atrica Speeds Restoration). This uses:

Both the tunneling and hardware mechanisms follow the protection model specified in the initial Metro Ethernet Forum agreements on Metro Ethernet Protection: Aggregation Link and Node Protection (ALNP) and End-to-End Path Protection (EEPP).

But not all carriers necessarily see Sonet-style sub-50ms restoration as essential for all circumstances. Says Epana's Seidman: “I don’t really think it’s that important to be down at 50ms. The reason is that a fiber cut that causes the core network to reconverge using Spanning Tree should be such a rare occurrence that, when it happens, the fact that it took a few hundred milliseconds to reconfigure is really the least of your problems as a carrier.”

He points out that the last-mile link to a customer is actually the weakest link in a network, as it is much more likely to fail than the core network – and so, when evaluating network failure performance holistically, 100ms or 200ms restoration times in the core network are acceptable.

Virtual private networks (VPNs) are crucial to Ethernet’s use in the metro because they are needed to support such key services as virtual LANs; so much industry effort is currently going into developing standardized Layer 2 VPN capabilities for the metro. From a service-provider view Layer 2 VPNs have a lot to recommend them:

But Layer 3 VPNs (basically, IP VPNs) are well established and have some attractive features. In particular, they are great for customers with all-IP traffic in a network with simple routing that employs a variety of Layer 2 technologies to connect sites. One of the aims of Layer 2 VPN development is to match this convenience.

However, there is a snag. Ethernet VPNs supporting transparent LAN service require any-to-any connectivity among MAC addresses at different customer sites in order to form the single broadcast domain required by Ethernet’s mode of operation. This is topologically equivalent to a fully meshed network and leads to the big bugbear of all Layer 2 VPNs: In a full mesh, the number of links increases as the square of the number of connected end devices. Even if companies forgo networked coffee dispensers, the number of connected PCs, printers, and other devices in a large corporation can run to many thousands – and the logical links into many millions. Layer 2 scaleability is clearly a potential problem if metro Ethernet VPNs take off in a big way.

There are quite a lot of standards and proposed standards for metro VPNs, and Table 2 lists some of them. The only fully standardized and widely implemented one for Ethernet Layer 2 is 802.1q VLAN, but this suffers from a severe potential scaling problem:

“In theory the maximum number of VLANs that can be supported is 4,096, based on the number of bits available in the 802.1q header, but in reality it’s not that,” says Elie Seidman of Epana Networks. “Most of these switches really bog down once they are running above 3,500 VLANs. This problem will eventually be solved by migrating to MPLS.”

Various vendors have also adopted a double-tagging (or stacked-tagging) variation to 802.1q VLAN to improve scaleability, but these are still proprietary fixes. Double – or higher-multiple – tagging allows a single VPN to support multiple VLANs, but the total number of VPNs is still limited to 4,096, as it is fixed by the size of the outer tag. The 802.1q VLANs scaleability problem is also exacerbated by the metro core switches having to provide VPN switching, and therefore having to be aware of both the VLANs and the MAC addresses to which they refer – and there is a limit to the number of MAC addresses that core switches can store: So even double stacking VLAN tags will still have to reckon with the storage limitations of core switches.

The well known MPLS-based Martini Draft is pretty much a de facto standard, with perhaps nine or 10 vendors offering it (at least in principle), but is aimed at point-to-point applications. The Kompella–Lassere Draft is similarly based, although not so well advanced or supported; but it is being promoted as the solution for point-to-multipoint VPNs, and so transparent LAN service. Both of these are being developed by the IETF and use a double tagging scheme to obtain scaleability.

Table 2: Different Approaches to VPNs

MPLS encapsulation of Ethernet is widely seen as the crucial step for VPNs, service provisioning, and general QOS requirements. MPLS encapsulation is much more scaleable than the basic 802.1q VLAN, because the label tag is 20 bits, not 12 bits. Also, MPLS encapsulations can be stacked to give multiple VC label-switched paths (LSPs) that are in turn bundled in tunnel LSPs.

Inevitably, variants are also appearing on the proposed IETF standards as vendors grapple with the specifics of implementing metro Ethernet VPNs. Nortel Networks Corp. (NYSE/Toronto: NT), for example, has recently implemented a variation on Kompella that it argues overcomes not only the scaleability issue, but also two other VLAN problems:

Nortel's approach uses MPLS in the metro core only, with Ethernet at the edge – a hub-and-spoke topology, which gives N- rather than N2-scaling for the connection-oriented MPLS. The service provider’s edge device (PE – provider edge) is divided into two separate components – PE edge and PE core – connected by a dedicated native Ethernet broadband link (the Ethernet Service Network). The PE Edge is a low-cost Ethernet switch that is the Ethernet UNI between the service provider and the enterprise; it manages SLA enforcement and customer information such as MAC addresses. The PE Core is a large network switch that handles MPLS tunnels and encapsulates Ethernet traffic onto the MPLS core network.

“We free up the core network from the burden of having to handle hundreds of thousands and millions of MAC addresses,” says Ian Jones, IP core SE manager for EMEA at Nortel. “By dividing the problem into two we have the advantage of confining the circuit-oriented approach to the core, with the simplicity of Ethernet in the edge. We believe that we have the best of both worlds.”

Currently, however, the impact of the scaleability issue can be overstressed. Extreme's Madanahalli points out: “The people who have installed networks and implemented VLANs to basic simple 802.1q have not reached the scaleability limit as yet. A quick map is 100-Mbit/s of bandwidth charged at $5,000 a month – that’s about $240 million a year for the full 4,096 limit. We would know if people were making that kind of money – but they’re not.” Extreme itself has a Japanese utility customer offering Ethernet-to-the-home to over 2 million customers without scaling problems by using company’s proprietary double-tagging VMAN scheme.

TDM circuit emulation is another key capability being developed for metro Ethernet that uses native-mode transport. According to Frederick Olsen, system architect for Agilent's networking solutions business unit: “The point of developing a circuit-emulation mode for Ethernet transport is that, looking at the revenues from carriers, a lot are coming from private-line services. If you are a new greenfield carrier building up a network, it is really key to be able to provide private-line services or TDM emulation services as part of your offering. Even RPR products that are shipping today provide TDM emulation services.”

Essentially, circuit emulation is a matter of being able to pack and time-stamp the synchronous TDM frames into Ethernet frames in such a way that, when transported by the fundamentally asynchronous Ethernet native mode, the received frames can be unpacked into the original TDM frames while maintaining clock synchronization.

Several vendors have proprietary solutions, and there are companies such as TeraSync Inc. that specialize in synchronization systems for asynchronous packet networks. Apart from providing the transport of a synchronous clock by Ethernet frames, Atrica’s implementation can compensate for clock changes – for example, when packet drops are detected – and support hierarchical clock dissemination – for example, the source clock could be OC3 and the destination T1. For clock distribution, the system can take an external clock input (typically stratum-1), the line clock (derived from the line signal), internal loop timing (not typical in commercial deployments), or the network clock.

There is an IETF proposal under development (Vainshtein Draft, after Sasha Vainshtein of Axerra Networks Inc.) for a standardized general approach known as TDM Circuit Emulation Service over Packet Switched Network (CESoPSN) (see Zarlink Unveils Access Processor). This is an encapsulation layer for carrying TDM circuits over packet-switched networks. Initially it's aimed at transparent NxDS0, transparent N*xDS0 with CAS, unstructured E1/T1, and unstructured E3/T3; but it may be extended to low-rate Sonet/SDH circuits with some modifications. It uses RFC 1889 Real Time Protocol (RTP) for clock recovery and supports signaling between Provider Edge (PE) devices. CESoPSN itself does not provide any resiliency mechanisms. A related IETF proposal under development is the Malis Draft (Sonet/SDH Circuit Emulation Service Over MPLS [CEM] Encapsulation).

Proponents argue that effective circuit emulation over packet networks such as Ethernet and IP amounts to a revolution in network design and economics. Says Aran Ariel, TeraSync’s director of sales: “By using our IPSync solution, metro network builders and carriers will be able to offer their local end customers legacy and new services – data, voice, video, frame relay, etc. – using a single metro network based on inexpensive Ethernet- and IP-based switches, as opposed to a network using expensive ATM switches, VOIP, and video-over-IP gateways.”

However, there is still skepticism over circuit emulation. Says Karen Barton of Appian: “From the carriers that we work with, they are not yet ready to accept circuit emulation as an alternative in supporting their voice TDM services. The quality and standards aren’t there yet – those are the key issues.”

Few Ethernet carriers, have tried such an approach. Epana's Seidman notes, “While TDM-over-packet makes sense for a short period of time, it is extremely inefficient. Certainly, in carrying a T1 over an Ethernet stream you are going to need more that a T1 of bandwidth. For what we do it’s not interesting. If you are an Ethernet player that’s only doing Ethernet over fiber to a customer, and you have a small number of customers and a lot of bandwidth per customer – then maybe it makes sense. But then again, if you have a big pipe to a customer, putting voice over the pipe in the form of packets is trivial. The real issue becomes an economic one, where one has to consider the cost of the equipment to convert TDM to VOIP versus the cost of the equipment to do TDM over IP.”

Theory is one thing, practice another. So how is metro Ethernet faring in the real world?

The ongoing financial meltdown in telecom in the aftermath of the technology investment bubble has taken a lot of the lustre off the alternative and greenfield carriers, and a lot of these are promoting pure metro Ethernet.

Yet this is more a verdict on business plans and market conditions than on metro Ethernet itself, and industry interest remains very high. Multifarious carriers continue to launch and extend metro Ethernet services. Here are some noteworthy examples:

Various forms of metro Ethernet are now in use worldwide. Major U.S. incumbents, such as AT&T Corp. (NYSE: T), BellSouth Corp. (NYSE: BLS), and Qwest Communications International Inc. (NYSE: Q) offer a range of metro Ethernet services for major and small enterprises.

In South Korea, KT Corp. is offering 1-, 5-, and 10-Mbit/s last-mile optical Ethernet broadband services to 80,000 homes in apartment blocks, under the Ntopia banner, using hardware-based rate limiting and accounting. The Canadian cable operator Videotron is using Ethernet access to offer Internet access and transparent LAN service to business customers. In Europe, the Swedish carrier Utfors AB is running Gigabit Ethernet to offer a range of business data and multimedia services in the Nordic region.

The more sober market conditions have swung attention back to the incumbents. These are becoming increasingly dominant in the metro as a result of the CLECs’ difficulties, and they have the resources to set the pace for metro Ethernet development. But this complicates technology choices because of legacy and regulatory issues.

The basic metro choice remains between Ethernet Over Dark Fiber (EOF), including passive CDWM/DWDM implementations, and Ethernet Over Sonet/SDH (EOS), although Ethernet Over DWDM (EODWDM) – that is, exploiting switched wavelengths – might be considered as a separate future category. But the wide range of ILEC environments and circumstances leads to a lot of mixing and matching.

“We see a combination of all the technology approaches in our solutions, to be honest,” says Nortel's Ian Jones. “Every ILEC customer has some degree of regulatory involvement that prevents it behaving in a monopolistic fashion, and this affects the choice of solution.”

Fiber cost per kilometer is one such key factor that can be affected by regulation. For example, the incumbent may have a subsidiary data service provider that is restricted on the price at which it can buy fiber from its parent – but the rules can vary from country to country. So for one incumbent an EOF implementation could be completely uneconomical, whereas the reverse might be true for another. For alternative operators there are the additional complications of the types of network mix they operate (access, metro, regional, and long-haul) and the availability and type of fiber.

So there is not really a typical metro Ethernet setup. But there are some emblematic types emerging that can be used as first-stab templates by different types of carrier and service provider: