MPLS Spurs Metro Ethernet Debate

Last week the Metro Ethernet Forum met in Boston to hash out a draft of technical specifications for deploying Ethernet in the metropolitan area network. The forum addressed areas that are key to making a more heavy-duty version of Ethernet for metropolitan networks: Ethernet Service Definition, Metro Ethernet Reference Model, Ethernet QoS framework, and Ethernet Protection (see MEF Touts Progress).

The issue of "protection" or network resiliency is fast becoming a major focus of debate.

What's surprised some analysts is the extent to which the proposal goes beyond simply aiming for traffic restoration within 50 milliseconds, the standard for legacy Sonet gear. Instead, the draft proposed by the MEF specifically states that MPLS (multiprotocol label switching) should be used to meet this goal. Analysts following the developments of metro Ethernet find this troublesome.

"I thought that their purpose was to define service levels so that carriers could create service-level agreements,” says Scott Clavenna, director of research at Light Reading and president of PointEast Research LLC. "I don’t think their role is to develop a method for doing protection. That’s the job of standards bodies like the IETF and the IEEE."

The meat of the controversy lies in the fact that for Ethernet to be accepted by service providers as a replacement for Sonet, it must provide the same quality and meet all the same standards as Sonet. This is especially critical if Ethernet ever hopes to replace Sonet for carrying voice traffic, for which 50 millisecond recovery times are a must. But this is easier said than done, because Ethernet was designed for local area networks, not large metropolitan networks.

Some service providers argue that MPLS is too complicated to be used for resiliency. They say that protection should be done at lower layers of the ISO stack to reduce complexity.

“I think that using MPLS for protection could be stretching its use,” says Eugene Pisman, a founder and the chief network architect at IntelliSpace, a metro service provider offering Ethernet data services. "It costs hard dollars to provision and buy the hardware to do MPLS. For example, from Riverstone or Extreme, you’d need to buy special blades Then sometimes they are implementing different versions of MPLS. You may get them to interoperate, but it will take work and will probably cost a lot to do.”

Clavenna and Pisman suggest other ways for handling resiliency. Resilent packet ring (RPR) technology is one way. The Resilient Packet Ring Alliance has been working with the IEEE 802.17 working group since last spring to come up with a standard for this technology (see RPR Divided). RPR, which has not been finalized yet, plans to develop a new MAC layer that will provide protection for all ring-based transport, including Ethernet.

Clavenna says that introducing a new signaling protocol like MPLS into the network could cause more problems than developing a new MAC to handle restoration.

But Nan Chen, chairman of the MEF and director of product marketing for Atrica Inc., an Ethernet equipment provider, argues that RPR would require service providers to buy new boxes, which he says will be expensive. He also says that disagreements within that standards body have made it difficult to predict the outcome of RPR (see RPR: RIP?).

"RPR is great for the future," says Chen. "When it’s complete we can take a look at it. Right now, no one is sure if it will be designed for voice traffic. Cisco’s version of DPT isn’t designed for that. Carriers want something they can implement now.”

Pisman agrees that RPR may be too expensive in some instances. He suggests using two technologies: RPR for the metro core, and rapid spanning tree for the access network. Rapid spanning tree is derived from the spanning tree protocol that has been used in local area networks for years. The new, enhanced protocol, which is currently going through the standards process in the IEEE, reduces recovery time to between one and five seconds, down from 30 seconds. While this is still not good enough for voice traffic, it is a huge improvement for service providers looking to provide real-time data services to customers.

Some Ethernet equipment companies are even improving the new rapid spanning tree protocol with one that they say can restore traffic even faster. Riverstone Networks has developed a technology it calls Rapid Ring Spanning Tree, which reduces recovery time to under 400 milliseconds (see Riverstone Rings Rapidly).

Riverstone announced today that France-based CompleTel Europe and South Korean-based Dacom are deploying Riverstone routers with rapid ring spanning tree turned on, to expand their metro Ethernet service offerings.

But the fact of the matter is that neither rapid spanning tree nor rapid ring spanning tree have gotten recovery times down to 50ms, the standard recovery time of Sonet-based voice networks.

“Right now there is nothing available that gives service providers 50ms restoration for Ethernet,” says Chen. “But using the MEF framework they will be able to.”

Still, other companies that are members of both the MEF and RPR Alliance say there is room for all of these technologies.

"MPLS wont solve every problem over night," says Gady Rosenfeld, director of strategic marketing for Corrigent Systems. “There are good things about it, and it is good to use for certain things like providing protection at higher layers, like between ring meshes. But RPR can be easier, cheaper, and simpler than MPLS for protection at lower layers. We plan to use both.” — Marguerite Reardon, Senior Editor, Light Reading
HarveyMudd 12/4/2012 | 7:25:30 PM
re: MPLS Spurs Metro Ethernet Debate MPLS is not likely to be deployed by the carriers here in the US or abroad. Some carriers may deploy it in certain parts of the for the purposes of traffic engineering and provoiding redundant routing.
linkv 12/4/2012 | 7:35:09 PM
re: MPLS Spurs Metro Ethernet Debate Gladys... good points, with one exception, "RPR
has a long way to to achieve even sub-second..."
I beg to differ; we've been shipping for almost
a year and consistently demonstrate 10-12ms
protection for TDM and packet data. That's live,
real customer traffic.


Luminous Networks, Inc.
rjmcmahon 12/4/2012 | 7:36:11 PM
re: MPLS Spurs Metro Ethernet Debate It's ironic that many in this industry don't get that "connection-less" is the 'thing' with internetworking - not ATM emulation (on routers or L3 switches).

For a large distributed information system, which talks packets, pcm samples, mpeg frames, scsi blocks, sometimes a connection is need and sometimes it is not.

Agreed 100% that the proprietary boundry is where the man made battles of differentiation will occur. In the case of the RRs, market forces caused the locomotives to be limited to their proprietors tracks while the box cars were standardized.

rriicc 12/4/2012 | 7:36:29 PM
re: MPLS Spurs Metro Ethernet Debate MPLS/VPN (2547b) across AS boundaries, across ISP networks; that's the scaling problem with MPLS as a VPN technology. The scalability is not about how many VRs or "mpls-routes" a PE can carry.
MPLS is in this sense the same thing as ATM and FR: it's connection-oriented.
MPLS is useful for TE within an AS - just like ATM. And TE is basically patching the lack of load and QoS sensitive routing protocols.
MPLS/VPN doesn't give any QoS for an individual VPN without diffserv (EXP..) and TE(MPLS). That is, if the LSPs aggregate multiple VPN traffic flows, PE-to-PE, which is the main proposal.
The security (P=private in VPN) is same as ATM (private pipe), who are happy with that when cryptographic security is available at fairly low cost? Running IPsec (tunnel mode) in MPLS/VPN is really silly - it's putting a VPN inside the MPLS/VPN.
It's ironic that many in this industry don't get that "connection-less" is the 'thing' with internetworking - not ATM emulation (on routers or L3 switches).
Gigaboy 12/4/2012 | 7:37:24 PM
re: MPLS Spurs Metro Ethernet Debate Great concept. Shall we start with multipoint TLS with in the metro or inter metro.
Man (no pun intended)like a bad girlfreind this protocol just grows on you.
drag 12/4/2012 | 7:37:34 PM
re: MPLS Spurs Metro Ethernet Debate MPLS will definitely have its place but primarily as a service enabler and L2 scalability enhancer (eg. L2 transport stuff)

MPLS VPNs do scale to realistic limits for 99% of VPN requirements. I mean, most edge devices will support around 200k+ routes total, that's more than enough for most apps. Again, apply some realism to the situation, how often would you really have the full table + 100000's of VPN routes on the same PE router? This whole scalability argument of MPLS VPN is quite sad... what's the best answer so far? Oh yes lets return to the wonderful world of L2 with all it's inherent *grin* scalability...

The complexity is not really related to MPLS directly it applies to QoS and TE in general and equally applies to IP, ATM etc. Whatever way you cut it these things are inherently more complex no matter what the protocol.


dark_light 12/4/2012 | 7:37:35 PM
re: MPLS Spurs Metro Ethernet Debate I do still see a place for mpls in the man and edge arena, especially for the VPN arena. While it may not scale well to huge porportions, it does allow for flexibility that other technologies do not, especially in areas of mixed topologies and multiple users on the same network. The biggest hurdle is it requiring new equipment, and you're right, why put a complex tunneling and qos system in when you can get your bandwith nirvana in the near future.
netskeptic 12/4/2012 | 7:37:38 PM
re: MPLS Spurs Metro Ethernet Debate to drag.

It will be real nice to keep separate networks for voice, it is quite possible that all attempts to have unified networks fail and this will be the only available choice. However, I suppose it will not happen in the next 5-10 years.

It will be nice to have no pipe throughput problems and hence no QoS - just 2 levels of diffserv and big buffers. Again, it is quite possible that this will be the end of it - data only network with abundance of both bandwidth, siwtching capacity and buffers. And again people now are serious about WRED - I suppose it is a clear indicator that there are years and years until bandwidth abundance nirvana.

And guess what - I do not see place for MPLS in the picture, which ever way the development will turn. If there are no bandwidth problems - simple
forwaridng will do it, if there are bandwidth problems MPLS is not equiped to help anyway.

BTW, I am observing the saga of IP/TAG/MPLS switching since 1995 and the persistence of the people who are trying to find a problem which could be solved by this technology does not stop to amaze me:

1. It was intended to solve forwarding problems at DS3, fast ether and OC-3. Hardware improvements in forwarding engines made it completely irrelevant.

2. It was supposed to solve VPN problems, it happen that MPLS based solution would not scale.

3. It was supposed to provide QoS and VoIP nirvana, it happen that DiffServ would the only available service with substantial problems on inter-carrier borders.

4. It was supposed to provide fast switching to
the redundant path, SONET already does it for years and years.

5. Now we are going to run ethernet on top of it - frankly I cannot think about more ridiculous idea - I suppose this signifies the logical end of the road for IP switching.


flanker 12/4/2012 | 7:37:38 PM
re: MPLS Spurs Metro Ethernet Debate The merits aside, the simple fact of the matter is that MPLS is up, running and being proven (or refined) every day. Cisco has already published off-the-shelf technical literature.

Where is RPR?
drag 12/4/2012 | 7:37:41 PM
re: MPLS Spurs Metro Ethernet Debate why not use both RPR/802.17/SRP/DPT/whatever + MPLS or Ethernet+MPLS depending on your situation?

MPLS is mostly a service driver these days in my opinion.

You're really trading off whatever 50ms means to you versus cost and complexity of MPLS-TE protection mechanisms. Either way, keep your Ethernet in the last mile to the customer to keep the CPE cost/complexity down. No need for STP on this single L2 hop. (also ask yourself how much fibre is really in rings between customer buildings?? Most is star wired back to the CO along the same route the copper is in or if the local utility was smart they wired it back to the local utility point). From a cost perspective RPR fit between POPs for the most part, unless heaven forbid we get a standard and economies of scale start kicking in...

On the topic of QoS, now even with someone having only 2 queues and someone else having 8 queues who cares at 10G speeds? Low latency is low latency everything else is everything else... we're not still having "time to clock a packet to the wire" or "how deep is the pass through buffer" arguments are we? How many of us would actually care to configure more than 2 QoS classes anyway when we have 10G speeds? Or even OC48 for that matter? Rememeber that one of the attractive features of Ethernet is that you get the large extra bandwidth for the same price or less, to remove complexity (like QoS) from your network. I would argue that 2 priorities are probably enough. Now someone will probably try and explain to me the difference between ATM CBR and VBR-rt or why we had to have a 6 bit field for the DSCP because 3 wasn't enough...

Face the fact that we will *never* shoehorn legacy TDM voice into packet based networks in the true spirit of if it aint broke don't fix it.

Many of the comparisons of Ethernet/RPR/MPLS and the highly abused layers of the OSI stack remind me of the same arguments between SONET/ATM/MPLS/IP - apples and oranges everwhere.

Wouldn't it be great if Cisco release SRP on Ethernet looking/priced interfaces instead of bungled into SONET!!! That would throw a few writers into a tailspin!!

I digress...


rjmcmahon 12/4/2012 | 7:37:42 PM
re: MPLS Spurs Metro Ethernet Debate Do you want a fail over protocol that has to message every 50ms?

A fail over protocol seems to me a distraction when considering demand (or congestion) driven circuit and path provisioning and creation, particularly considering the capacity optical networks bring to the table.

Argueably, a fail over protocol could be useful if the business model were based on access but seems not if based on transactions. And its hard to see highspeed optical networks terribly relevant in access based business models, in my opinion.

gea 12/4/2012 | 7:37:42 PM
re: MPLS Spurs Metro Ethernet Debate Well, strictly speakiong MPLS is not itself a protocol. It relies on the existence of a whole bag of protocols. So it can be as smart or as stupid as you want it, depending on what protocols you have operating with it.

As for MPLS not being able to control the physical layer, that's not exactly true. With GMPLS, networks will have the ability to turn on physical layer protection mechanisms via the control plane protocols, such as LMP, LDP, UNI and NNI.

As for poster's comments, I'm happy to see someone who seems to know a little bit about something posting here. His comment about FEC being more important for OSNR than for optical power is a good one I often forget. I don't, however, don't agree with a couple of his other comments about SONET:

1) No carrier-to-carrier connectivity? What do you mean? SONET NEs interoperate all the time between carrier networks, which was one of the primary rason detres of SONET. (Inter-vendor meets are also common, contrary to common belief).

2) Internetwork PM not possible? Not true--SONET networks will not terminate the path between networks, so B3, J1, and all sorts of other bytes will work just fine. Theoretically, if I have a router on one end of the country, I can even determine what BER its peer on the other end of the country sees via the REI-P byte.

As for GFP+G.709...well, I'm tired so maybe I'll try to answer tomorrow.
dark_light 12/4/2012 | 7:37:44 PM
re: MPLS Spurs Metro Ethernet Debate The problem is MPLS itself is not a smart protocol. Its takes a real routing protocol like RSVP to make MPLS a viable solution for the MAN. MPLS doesn't set up secondary routes and such, its the job of another protocol. There is no inherent way in mpls for it to know a tunnel is down other than some form of ping. So to have such fast fail-over times such a protocol needs to be send tons of hello messages and clog up the network in general. This is clearly the main problem with mpls, and worsens as the network gets more and more complicated. Other than that MPLS is ideal as its highly flexible.

Do you want a fail over protocol that has to message every 50ms?
skeptic 12/4/2012 | 7:37:44 PM
re: MPLS Spurs Metro Ethernet Debate
The problem with the approach is that its based
on the idea that overlays (MPLS) solve all
the problems in the network. You end up trying
to solve physical problems at higher layers
in networking. And in the long run, that isn't
going to solve the problem.

If you can't do 50ms restoration on ethernet
without MPLS, its not really credible to think
that MPLS is going to solve the problem in
software. Your building soft restoration on
top of a physical media that doesn't support
restoration well in the first place. And if
it can be done with MPLS, there is nothing
to prevent other non-MPLS solutions solutions
to the problem.

jhakaas 12/4/2012 | 7:37:45 PM
re: MPLS Spurs Metro Ethernet Debate > Finally, G.709+GFP (Generic Framing Procedure) > might make a potent little AIDS cocktail for
> some of what ails the network.

Can you explain the above?

poster 12/4/2012 | 7:37:45 PM
re: MPLS Spurs Metro Ethernet Debate good points gea. but SONET does have serious limitations. no TCM = no carrier-to-carrier handoffs. no PM across multiple carrier networks. DW has 8 layers of PM in TCM.

and SONET was only designed with one wavelength in mind. pretty simplistic an incapable of managing anything but a circuit based, single-wavelength ring. given that DWDM is the de-facto transport technology now in fiber networks, I see a limited lifetime for SONET as the underlying transport protocol.

turns out the real advantage of FEC is not distance, but OSNR. significant difference.

the way I see it you are not limited at all by G.709 to OCh protection only. it just becomes an option if you choose. any higher-layer packet or SONET protection method can still be used if desired. just disable the OCh-level protetion. I strongly beleive SONET, along with MPLS, RPR, 10GbE will all transition to be service layer protocols, not transport. that is carriers will build these rings on top of G.709 wavelengths.

SONET/SDH will now be used for circuit services, basically muxing/grooming/switching anything < OC-192. the packet protocols will be used for flow-level muxing/grooming switching any packet-based services < 10Gbps. and with packet-over-G.709, now you have all of the lacking transport-layer functions like optical protection at 50msec if you so choose. an guess what they both operate on a single G.709 transport network. becuase it offers the highest level of transport control, exceeding SONET/SDH, it becomes the lowest common denominator.

yes you are 100% correct about provisioning. G.709, like SONET, really makes no effort to automate provisioning in and of itself. MPLS/G-MPLS will be incorporated to do that. the key difference is, unlike SONET, it has been designed with that in mind.

don't worry I'm not religious, but this just seems so simple.

gea 12/4/2012 | 7:37:46 PM
re: MPLS Spurs Metro Ethernet Debate Ah, Poster...you ask a loaded question!

Why wouldn't anyone NOT use G.709? My canned answer is, "because we already have a perfectly fine digital wrapper. It's called SONET."

Surely you must be thinking that this is a preposterous statement, but if you look at the digitial wrapper standard and compare it to SONET, I think it makes some sense.

When I say "SONET", I don't mean circuit switching, I mean SONET framing, possibly in the form of large,concatenated (or virtually concatenated) payloads. Increasingly, I think we'll see lots of optical layer formats grow a SONET-framed version, and 10GbE is only the first.

Although I don't have time to drill down into lots of details, let's just say that the use of G.709 REQUIRES an "output" transponder before handoff to the client layer, which is uneccessary if the client is SONET frame-friendly. Sure, G.709 has got FEC, but I really only need that for long haul networks, or 10Gig in the metro which I can often avoid by using the in-band (i, within the SONET overhead) OC-192 FEC.

In addition, G.709 will often limit you to optical channel-type protection...this is probably inadequate for the flow-level multiplexing done in MPLS networks.

Finally, G.709 does nothing for provisioning. The MPLS/GMPLS vision is to allow control plane protocols to set up and tear down data/optical/TDM paths, and possibly also protection. With G.709 I still need the control plane protocols unless i want to do manual provisioning.

That said, there are things I like about G.709, particularly in networks that run lots of different rates and protocols. I'm also starting to like the various flavors of FEC that can be utilized. Finally, G.709+GFP (Generic Framing Procedure) might make a potent little AIDS cocktail for some of what ails the network.

So don't ya' go gettin' too religious on us Poster.
gea 12/4/2012 | 7:37:46 PM
re: MPLS Spurs Metro Ethernet Debate GladysNight: Great quote, and very funny.

The fact that the Metro Ethernet forum has "agreed" on using MPLS probably reflects the fact that, in one way or another, the MPLS control plane protocols are going to play a very important part in a lot of networks in the future, either through GMPLS, MPlamdaS, or MPLS (in the context of layer 3 or here, layer 2).

I think there's enough work going on collectively on these protocols that there will be a lot of nutritious "scraps" falling from those tables, so that a coherent protection approach in the context of Ethernet should be straightforward. I think it likely the MEF pretty muich understands this, so decided to latch on the this wave of work rather than start from scratch. Sorry--sound like a smart decision to me.
poster 12/4/2012 | 7:37:46 PM
re: MPLS Spurs Metro Ethernet Debate which transport technology will win in the metro? none. it's a moot point. MPLS is a traffic engineering protocol, not a physical layer transport method. RPR is great for packet rings, not for circuits/voice. same for 10GbE. additionally, aggregate ring bandwidth for both of these top out your highest rate interface (10Gbps for RPR, maybe faster in the future - 10Gbps tops for 10GbE)

why would a carrier standardize on transport protocol that's optimized them to one traffic type, in this case data, puts a ceiling on capacity, and lacks the required protection failover? why do so many people now want an alternative to a strictly SONET-based network? for the same reason.

who wins? digital wrapper-based DWDM transport platforms. all of the protection capabilities of SONET combined with mappings for all data protocols. G.709 DW exceeds SONET PM capability as well. no ring bandwidth limitations. designed for physical layer transport. carriers will build RPR and/or MPLS and/or 10GbE packet *AND* SONET rings on top of G.709 DWDM networks.

and guess what? the technology is standardized and available on chips NOW. someone please explain why they think this WOULDN'T happen. (aside from the fact that some soon-to-be defunct carriers only plan to offer data services)

tintin 12/4/2012 | 7:37:48 PM
re: MPLS Spurs Metro Ethernet Debate As one of my colleague would put it (with a lot of, obvious, humor): "MPLS will save the world"..., it might not save the world, but it does simplify carrier architectures significantly.
The reason is that it is medium-independent.
This is also why it is in-line with MEF's positioning statement which leaves physical media out of its realm.

My $0.02.

gladysnight 12/4/2012 | 7:37:48 PM
re: MPLS Spurs Metro Ethernet Debate You make some good points metroman.

My reservations about this whole discussion are more about having seen other candidate technologies for "silver bullet" and having watched them regularly fail to deliver networking nirvana.

In spite of the sense of much of your post, I believe that there remains an element of the "one ring to rule them all, one ring to bind them" mentality in much of these marketecture efforts.

The whole reason we have a 7 layer model is that horizontal layered architectures fundamentally work, they make networking easier, they aid interoperability, and they ease migration of specific layers.

I believe that trying to collapse functions of multiple layers into a single technology is more or less doomed to failure.

From my pov protection switching is a layer 2 function and should stay there. In SONet/SDH we already have a working link-level protection mechanism, and it works for pretty much all traffic types. RPR has a long way to go to achieve even sub-second times, and MPLS still isn't anywhere as interoperable as ethernet, which it would need to be to make an ethernet based end-to-end MPLS solution workable. Even when it gets there, there remains doubt in my mind that it will work as advertised, but I am sure that a lot of operators will spend a lot of money finding that out.

Just as they did to discover that ATM is not networking nirvana, IP is not packet paradise, and so on and so on and etc . . . . .

I'd prefer to see development of a simple and cost effective SONet/SDH blade for metro scale (GigE and 10GigE) ethernet switches. Possibly a v.short haul implementation using lowest cost components to achieve 3 to 5 kilometre spans at a maximum.

This would interface directly into existing transport networks, require few if any new skills, and ideally would be managable from existing management applications.

For many operators that would be Christmas in July.

metroman 12/4/2012 | 7:37:49 PM
re: MPLS Spurs Metro Ethernet Debate You should not take the view that MPLS is just for redundancy. It is designed for a great many service orientated solutions that will add value for SPs and drive costs lower over Ethernet.

The issue here is to compare the Metro networks of today with the incumbent providers who are offering Frame and ATM services as their predominant offering. If you create a network topology that has the cost advantages of Ethernet and overlay ot with a common control protocol such as MPLS you have a perfect match. MPLS will act as an enhancement and a substitute for some of Ethernet's failings. Enhancing VLANs by the creation of Draft-Martini services and replacing spanning tree (and some IGP convergence issues for that matter) with hot standby and fast re-route LSPs. Draft Martini tunnels can also be engineered for carrying ATM and Frame traffic over the underlying Ethernet topology. All of this means that you have a single network to operate and a network deployed based on low cost components. (even the additional MPLS hardware is cheap compared to having 3 networks.)

So to say that MPLS is just there for protection is not true, it is there to deliver a number of other services while enhancing the way Ethernet deals with failures. If you rely on the lower layers to protect your data services you will more than likely create cost and cause IGP issues which will negate the 50ms nirvana.

RPR is so split that I think by the time it arrive's in a standard form it's speeds will be too slow and will only be useful as an access technology and only if you run MPLS over it so you have an end to end control protocol.

Rapid Spanning Tree in the access and MPLS in the core is great, but have you tried to deploy it? It makes more sense to have a common control protocol so you can fully control opex and SLAs.

Sign In