I’ve been talking to a lot of people about optical Ethernet lately, trying to determine just what it will take to create a profitable carrier service around this “new new thing.” Along the way, I met Roy Bynum, formerly of WorldCom Inc. (Nasdaq: WCOM), now an independent consultant working out of Texas.
Roy was the instigator of the 10-GigE WAN PHY, as well as a co-author of the X.86 LAP-S Ethernet-over-Sonet protocol and the X.86 Ethernet Flow Control Specification. He also was the architect of what is now commonly referred to as the Ethernet Private Line, which is, by most accounts, the only truly successful (read: "profitable") Ethernet-based service from a carrier today.
Roy is a big advocate of Ethernet. He sees it as not only the “next big thing,” but as the likely successor to the throne currently occupied by Frame Relay in the data networks of service providers – and by T3 on the transmission side of the house. This is big. It means Ethernet ultimately emerges as the winner in both transport and switching, just as it has within private enterprise networks.
Roy’s history lesson goes this way: A little less than a decade ago, Frame Relay overtook X.25 because it was cheaper for the service provider to operate and cheaper for the customer to own. Frame Relay could take a user up to 1.5 Mbit/s easily, which was big bandwidth at the time. Frame Relay service facilities are much more scaleable than X.25, thanks to encapsulation of customer frames in Asynchronous Transfer Mode (ATM) or Internet Protocol (IP).
That virtual switching fabric also allowed the service providers to distribute the Frame Relay access gateway facilities, which meant that the Frame Relay customer access did not need to be backhauled to large centralized switching points, making the local loop access a lot less expensive.
It also made it easier to build a redundant, diversely routed network architecture to support the interconnections among Frame Relay gateway facilities in different parts of the country. As history proved, Frame Relay was actually a better service, a better product, with overall better cost of ownership for both the service providers and the customers. Frame Relay opened up new markets for packet services by providing long-haul point-to-point and point-to-multipoint connectivity without the distance charges of private line or the volume charges of X.25.
For a while, people thought ATM would supplant Frame Relay once bandwidth demands from end users grew, but it didn’t: too expensive, too complex, violating the rules of low cost of ownership for carrier and customer alike. But along comes Ethernet, which today is appearing as an interface on just about every piece of data and transport networking gear in the service provider, after completely overtaking the enterprise.
Where Ethernet has already started to make an impact through simple Ethernet private lines (Ethernet mapped directly to Sonet channels), it’s worth taking some time looking at how Ethernet may just take over the packet services market as well. The form it takes may not be so different from Frame Relay, just faster and cheaper, and since Ethernet is based on its own set of standardized frames, it may make sense to just call this new service and infrastructure Ethernet Frame Relay.
The key, Roy argues, is Multiprotocol Label Switching (MPLS) encapsulation of Ethernet. No funny business with customer frames is allowed, giving users the utmost sense of security and giving carriers a way to carry that traffic without having to process it at anything other than Layer 2. Since the introduction of Frame Relay there have been several changes to the Internet Protocol standard that allows Layer 2 switching of end-to-end flows. This, in effect, creates a distributed switching fabric at the data-link layer, where before it was at the network layer. This should provide a much more stable and cost-effective way to implement the same type of encapsulated frame switching that was done for Frame Relay ten years ago.
Ethernet, the most commonly used protocol in the world, has matured in the last ten years. Legacy-free service providers experimented with providing direct Ethernet VPN (virtual private network) access to Internet services using Gigabit Ethernet as the transport protocol. Ten-Gig Ethernet has added a Sonet-like overhead to support managed optical wavelength services. The Institute of Electrical and Electronics Engineers Inc. (IEEE)P802.3ah Ethernet in the First Mile (EFM) Task Force is in the process of defining subscription network service-provider management support. Ethernet, instead of being only for privately owned enterprise networks, is now becoming a basis for building new types of revenue-generating data transmission services.
The advent of MPLS and EFM at the same time now makes it possible to provide the same kinds of networking services for Ethernet that were previously provided using Frame Relay. By using MPLS to encapsulate Ethernet frames, a distributed virtual switching fabric can be built for long-haul Ethernet point-to-point and point-to-multipoint links without having to pay long-distance private line charges. In addition, Ethernet/MPLS gateway switches can be distributed throughout metropolitan areas in large as well as small cities, making the virtual switching fabric even more distributed. This is exactly what is needed to create the next generation of wideband data transmission services.
MPLS has better QOS (quality of service) and bandwidth reservation capability than the older Internet Protocol that was used for Frame Relay. In addition to the ability to provide tiered bandwidth management to provide the same PIR/CIR (peak information rate over committed information rate) relationship, the use of Ethernet active flow control in the local access loop makes it possible to provide a more granular PIR as well.
Additionally, active flow control might be used to control the CIR/PIR relationship by using the MPLS forward and backward congestion control to generate a flow control message to the customer’s Ethernet switches on the ends of the access links. The active flow control message would slow the customer's transmission of Ethernet frames down to a level that prevents too much data loss during times of congestion. This will be a big help during peak business hours or at times when a fault in the transmission network reduces the service capacity of the distributed switching fabric.
With this, Roy Bynum believes, broadband access technology can be used to provide a wideband PIR for the narrowband CIR. Over a period of time, given the right market circumstances, it will provide an evolutionary migration path from wideband/narrowband packet service demand to broadband/wideband packet service demand and beyond. Where the need exists to use the existing copper local access infrastructure, 802.3ah is also defining an xDSL-based form of subscription network managed Ethernet. This wide variety of physical access standards will support the existing Frame Relay narrowband access as well as the growth to the high bandwidth of physical access.
See? That’s big. For enterprise customers without fiber ot the building, Ethernet-over-copper solutions get them on the network via a simple LAN interface. For those with access to fiber on metro optical rings, Ethernet access is already possible via Ethernet-over-Sonet or dedicated GigE links to service provider nets. Adding this encapsulation of Ethernet via MPLS is the final ingredient to taking Ethernet from an access service to a switched network service, where the real money is. Carriers haven’t ever been able to generate a lot of profits from packet services, just revenue. Something needed to come along and truly disrupt the cost of ownership of packet services to really get carriers serious about deploying broadband data services infrastructure. That means carrier spending, and that’s good for everyone.
------ Given that a typical OC48/OC192 has an operational bit error rate of about 1 in 1,000,000,000,000 bits, at about 380 to 400 bytes per packet, an IP routed network should see a data loss of 0.000001%, not 0.1%. Where are all of the additional errors coming from? ------
BER and Packet loss are two different things. The most frequent cause of packet loss on an uncongested network are re-routes due to human error, fiber cuts or equipment failure. A 3000km long OC48 might have a BER of 1x10-12 or so. But it only has an availability of less than 99.5% (That would be a neat trick actually, it's more like 99.0% when including electronics).
Now, say it's SONET protected and now has a 99.9975% availability. Now we're around 0.0025% for just the fiber that OC48 is on. Haven't accomodated the biggest factor, human error, or of course, hardware failure (lots of transmission equipment to light that fiber or packet gear).
Of course, on a congested network, all bets are off.
Steeler: If your buddies were so damn bright, they would have patented their ideas or at least found a VC to back them ________________________________________________
Part of the problem with working for a service provider, is that the service provider employer wants to own the idea. The idea, concept, technology, architecture, what ever, is often given to the vendors for free in exchange for the vendor building the systems to make the service product work for the service provider. In the past, very little technology was invented outside of the service providers because it was the service providers that came up with the exact solutions that were needed to address their issues. It was only recently; during the "bubble" that technology appeared to be invented outside of the service provider. If you remember, AT&T owned Bell Labs where much of the telephony technology was invented. A few "devices" were developed outside of that environment, but it is a small percentage of the whole.
Other than one or two people, I know of no one that has gotten any support from a VC for original technology ideas. In the past, VCs provided money to people that they knew more than they did to the technology ideas. I hope that I am wrong, but most of the VC money that I am aware of goes to data centric ex=vendor people, not the data transmission services people.
With the down turn of the telecom industry, a lot of the original thinking data transmission services people are going to be available to the VC community. In order for new markets to open up, it will require that level of original thinking, with the back ground of what it takes to make services work and what the technologies need to be to create sustainable service offerings, not the back ground based on the hype of the “bubble”.
It turns out that the services that came out of the “bubble” are themselves not as robust economically as we were led to believe during the “bubble”. The failure of Yipes is a good example. Frame Relay is not a product of the “bubble”. Frame Relay preceded the “bubble” and remains one of the primary revenue generation services in the business services market. The fastest growing service by connectivity percentage is DS3 private line. Ethernet as a lower cost of ownership technology maps well into that legacy market to open up new lines of services. There is still specific transmission service quality technology that needs to be developed for Ethernet to make that happen.
Roy, don't be so bitter. If you want "credit", go to a university. Most of the wealth in this business comes from selling a technology, not inventing it. There is no imbalance. If your buddies were so damn bright, they would have patented their ideas or at least found a VC to back them.
giles: secondly I think it is worth remembering that packet loss is a design feature of IP networks. That's how we tell sources to slow down in periods of transient network congestion. It is also the behaviour you'll see during routing transients. __________________________________________________
From what you are saying, there is no way in the world that an IP based data services network can ever achieve the data integrity of a TDM based data services network. An IP based data services network can not even achieve the data integrity of a flat Ethernet based data services network. In fact, it is not supposed to have the data integrity of a TDM based data services network.
It is no wonder that broad band data and other packet data services are primarily at narrowband bandwidths. It is also no wonder that the vast majority of high bandwidth business services are TDM based leased circuits.
Perhaps the IP protocol was a bad choice for data services network infrastructures, although at the time, there was nothing else available. (As I was one of those that helped make that decision, I am partly to blame for the mess that we are in.) The next question is, since we have to live with IP, how can IP be changed to remove the requirement for data loss? Perhaps we can go to UDP/IP over Ethernet instead of TCP/IP over whatever? (UDP does not have data loss flow control functionality that TCP does.)
giles: though it is hard for me to see how much of the jitter is my network and how much is processor scheduling in my network management boxes _______________________________________________
There are tests scripts that test for latency variance that are now available on the major data test sets that are available in the market. You can set one of these up at one end, with another at the other end and send data looping through your network over an extended period of time to give your average data jitter. Shorter tests can be run during "busy hours" and "off hours" to give the delta between a "loaded" and "unloaded" network. Also various routes can be tested end to end using this technique.
------ Given that a typical OC48/OC192 has an operational bit error rate of about 1 in 1,000,000,000,000 bits, at about 380 to 400 bytes per packet, an IP routed network should see a data loss of 0.000001%, not 0.1%. Where are all of the additional errors coming from? ------
first off I'd have to say that the figure I quoted was "< 0.1%". My network for one sees packet loss well below 0.01% - though the SLA is set at 0.1% so we don't risk breaking it :)
secondly I think it is worth remembering that packet loss is a design feature of IP networks. That's how we tell sources to slow down in periods of transient network congestion. It is also the behaviour you'll see during routing transients.
The source of this packet loss is most likely RED on the routers kicking in as buffers fill up during periods of increased load?
------ Data jitter (latency variance) should not be any greater than 125us (125 microseconds) for mixed size packets. The 125us is the buffer windowing of the SONET payload on the transmission interface. Because this is an external artifact, it should remain constant without compounding as the data traverses multiple elements. Data jitter greater than 125us is because of processing latency variance in the "routers". Most of the IP routed networks that I know of measure data jitter in the ms (milliseconds). ------
remember that memory is cheap. If your network jitter is > 125us then you can buffer to keep jitter constant over a given pseudowire.
I see sub-millisecond jitter on my network (though granted probably not as low as 125us - though it is hard for me to see how much of the jitter is my network and how much is processor scheduling in my network management boxes).
That is a LOT of hard work that these people put into something to have others, not even of their own company get the credit for and wealth from. ____________________
Many have done a lot of work and too few have received just rewards and recognitions for their efforts.
And from my viewing distance, which is far, the LVLT guys seem to have done one hell of a job. The market hasn't rewarded them for their work though the game is still early.
DISCLAIMER: I do have a position in LVLT and I will personally benefit if our stock price increases. (I say "our" because I do believe in that "stakeholder" stuff).
Vendor marketing people often do not want to be hampered by the realities of what it takes to prove in a sustainable service, or what it is that makes a sustainable service. Sometimes service company executives have made statements that “stretch” the reality of how much business one service or another is actually doing in order to make their company appear to be doing better than it actually is. When these to factors come together you get a quagmire of miss-information that can not credit work or ideas that are successful and sustainable.
==================
Translated: Vendors lie to carriers, carriers pretend to believe. Everyone lies to the VCs, who package the lies for the investment banks, whose analysts aren't smart enough to figure out that everyone's lying.
Stock gets sold to mutual funds in bubble market, who bask in "performance" numbers derived from hot IPOs. Smarter mutual funds get out by selling stock to individual investors who are the last to figure anything out.
rjmcmahon: The moral of the story is that applications should not rely on link level FCS for payload protection. ________________________________________________
The need for end to end path level FCS is the primary reason that MPLS should not stript the Ethernet FCS off when it encapsulates an Ethernet frame. By keeping the FCS that the customer originally put on the Ethernet frame, a true end to end error detection is achieved. Having the hierarchy of bit error detection will allow better problem isolation between the customer and service provider infrastructures.
rjmcmahon: The time is now to create long term value. Ideas are merely the seeds. Human effort and commitment to others is the next step required to grow those seeds into crops -- crops that can feed a society. ________________________________________________
What I was referring to was not just the idea of something, but the design of the architecture, the deployment of the proof of concept, the proving that the “preferred vendor solution” was not what was needed, the living with and the proving in of a concept and an infrastructure. That is a LOT of hard work that these people put into something to have others, not even of their own company get the credit for and wealth from.
Vendor marketing people often do not want to be hampered by the realities of what it takes to prove in a sustainable service, or what it is that makes a sustainable service. Sometimes service company executives have made statements that “stretch” the reality of how much business one service or another is actually doing in order to make their company appear to be doing better than it actually is. When these to factors come together you get a quagmire of miss-information that can not credit work or ideas that are successful and sustainable.
The blogs and comments are the opinions only of the writers and do not reflect the views of Light Reading. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
To save this item to your list of favorite Light Reading content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Digg
Del.icio.us
Reddit
