Internet Peering on Thin Ice?

A tiff over a peering agreement last week between Level 3 Communications Inc. (Nasdaq: LVLT) and rival ISP Cogent Communications Group Inc. (Amex: COI) has left some people -- especially customers of the two -- marveling at how fragile a thing the public Internet can be and how easily it can be compromised. (See Level 3 Tweaks Cogent and Cogent Tweaks Level 3.)

Level 3 interrupted a free, or "settlement free," peering connection with Cogent last Wednesday, which resulted in tens of millions of IP addresses effectively vanishing for many enterprises using either Level 3 or Cogent for Internet access.

How could that happen? Hunter Newby of telx Group Inc. says part of the answer is the way the Internet is set up. He notes that the Internet comprises ISP (and other non-commercial) networks that are loosely stitched together on handshake deals among a very select group known as "peering managers."

"IP peering is an interesting and delicate business,” Newby says. “It is made up of people and relationships that are rarely documented and have no remedy provisions, no real definition of breach, no damages provisions, etc."

In other words, it's business done on a handshake, with little legal documentation. That makes it easier to clip a peering connection, even if consumers will be negatively affected.

“How L3 peering managers can make a decision like that and just de-peer a network as large as Cogent is amazing,” Newby observes. He characterizes Level 3’s action as one of the biggest disruptions in the history of the Internet.

Of course, Level 3 doesn’t see it quite like that. “I completely take exception to that characterization,” says Level 3 COO Kevin O’Hara, adding that his company has de-peered many other networks in the past with “little or no” disruption in Internet service. (See EarthLink Uses Level 3.)

“Our goal all along has been to make sure no consumer or user of the Internet is impacted by the commercial issues happening deep within the bowels of our network,” he says.

But there could be a lesson here: Enterprises might want to investigate whether their ISPs have contingency plans in place so that access to the Internet is never put in jeopardy.

Who is really to blame for the events last week is now mostly lost in a fog of claims and counterclaims. Level 3 says Cogent has been sending more traffic over Level 3's network than they've been sending in the other direction, and that they should therefore pay a tariff. Cogent, meanwhile, insists that it has satisfied all of the various traffic volume and bandwidth metrics in Level 3’s peering requirements, and should be allowed to continue peering with Level 3 at no cost.

Cogent CEO Dave Schaeffer told Light Reading Wednesday that the real reason Level 3 clipped the peering connection last week is because it believes Cogent is pricing its Internet service at “below cost,” making it hard for Level 3 to compete.

"In mid-September I got a phone call from a very senior person at Level 3, and in that telephone call they said, 'We’re going to cut you off if you guys don’t either pay us or reconsider your pricing strategy and raise prices,' ” Schaeffer says. (See Cogent: King of Ports .)

Cogent sells Internet access for around $10 per megabyte, according to the company, while the Level 3 rate is somewhere between $25 and $60 per megabyte. (See Cogent Revenues Jump in Q2.)

Level 3’s O’Hara strongly denies Schaeffer’s claim. “I have heard this allegation before and we take it very seriously, and to my knowledge nobody from this organization has made that request of Cogent, and that’s as far as I’m going to go with that,” O’Hara told Light Reading Thursday.

O’Hara says after Cogent didn’t respond to several notifications that de-peering was imminent, his company assumed Cogent had made a comparable peering arrangement elsewhere.

When it became apparent that Cogent had not, and that many users were effectively cut off from large portions of the Internet, Level 3 reversed itself. Two days after the de-peering, the connection between the two networks was restored, and Level 3 says it will remain so until November 9.

In the meantime, Cogent must either strike a deal with Level 3 or find a comparable peering arrangement elsewhere. As of Thursday, the two companies hadn't talked.

Historically, the U.S. Congress and Federal Communications Commission (FCC) have left it to the network operators to work out their peering arrangements. But some worry that if peering disputes between IP networks become more common, Congress and the FCC may be compelled to act.

“When Congress sees something like this happening and it looks as if consumers are the ones suffering, they naturally want to jump in and solve the problem,” says Staci Pies of the Voice On the Net (VON) Coalition. “The question is, whether jumping in and solving the Level 3/Cogent problem, if in fact there is a problem that needs to be solved, does damage elsewhere.

“We are very much in favor of Net neutrality principles,” Pies adds, “But we are not in favor of interconnection obligations and standards for IP-to-IP networks.”

Former FCC chief of staff and current Legg Mason Inc. analyst Blair Levin agrees with Pies that it would take far more than just one peering dispute to push Congress or the FCC to act. But Levin also points out that the current atmosphere of consolidation and convergence in IT and telecom may increase the risk of such disagreements in the future.

— Mark Sullivan, Reporter, Light Reading

Page 1 / 3   >   >>
Mark Sullivan 12/5/2012 | 2:57:26 AM
re: Internet Peering on Thin Ice? Cogent's Schaeffer says the whole dispute is really about Cogent's bargain basement access prices. Is there something to his allegation? Will this end up in court?
fiber_r_us 12/5/2012 | 2:57:25 AM
re: Internet Peering on Thin Ice? I can't really imagine this going to litigation.

Having been involved in this stuff in the past, the characterization that peering is "nothing more than handshake" agreements is not accurate. There are most certainly written agreements in place.

The whole issue has more to do with what is more correctly referred to as SFI (Settlement Free Interconnect). SFIs are agreements between two Internet providers where they share the cost of interconnecting the two networks (usually in multiple places throughout the country with really big 10G pipes), but don't charge each other for traffic passed over those interconnects. Peering can also take the form of paid-peering when the parties are not on equal footing. All SFI agreements allow either party to de-peer (with notice) when the SFI arrangement is no longer beneficial to one of the parties.

SFI only makes sense when it is mutually beneficial to both parties (hence the term "peer"). Whenever it becomes lopsided, then the party on the losing side of the agreement is likely (even compelled) to initiate de-peering.

This is a perfectly *normal* thing! Peering and de-peering occurs on a regular basis in the industry and usually goes completely without notice to the users of the Internet. What made this situation unique is that, when de-peering occurs, at least one of the players is essentially forced to:

- buy capacity from someone other than the original SFI peer that connects them with the original network (called GǣtransitGǥ), or
- Convert from GǣSFI peeringGǥ to Gǣpaid peeringGǥ with the original party

One of these is required in order to restore connectivity between the two networks. So, who should do this?

The Internet has grown-up in a way such that there are those Internet backbones that have customer bases that generate an equal amount (more or less) of traffic to/from other backbones of similar usages patterns. Large backbones that fit this pattern typically form SFI with each other and are technically referred to as Tier 1 peers. These Tier 1 backbones form connections with other Internet network players with:

-Paid peering, or
-Transit through other networks

Since Cogent already pays for transit access to numerous other Tier-1/SFI networks (Sprint via Verio for example), Cogent isnGt a Tier 1 backbone as defined above. IGm sure they market themselves as a GǣTier 1Gǥ, but, hey, thatGs marketing! Also, since CogentGs margins are very thin, they probably were not interested in spending more money on either paid peering or additional transit capacity.

Since Level 3 is a Tier-1/SFI based network, all other players connected to Level 3 either are other Tier-1/SFI networks or pay Level 3 for transit.

Cogent is playing a "game" with the Internet and trying to use the uninformed public to try their case. The rate that they charge for their service is, indirectly, part of the issue. What Cogent would *like* to do is "cherry pick" all of the high-capacity Internet server oriented customers and give them really cheap bandwidth and dump the traffic onto other networks at the closest location (so they don't have to use their own expensive resources to haul the traffic). The result of this type of business approach is that it makes SFI difficult (impossible) as you are simply abusing other networksG capacity in order to support your cheap rates! One of the signs of this behavior is the asymmetrical traffic on SFI links.

Level 3 is only protecting its own business when it initiates the de-peering (and they had given Cogent ample warning). That is, Level 3 should not have to spend its resource supporting CogentGs business model. Cogent, not being Tier 1, failed to acquire alternative connectivity to Level 3 during the notice period (undoubtedly hoping Level 3 would Gǣback downGǥ knowing that customers would be affected. But, though Level 3 initiated the disconnect, it is not Level 3 responsibility to make CogentGs network GǣwholeGǥ.

I suspect other Cogent SFI peers that are in this situation are watching this very closely to decide whether it is worth the PR impact to de-peer their lopsided Cogent connections. Cogent has everything to loose! If they stop getting their Gǣfree ridesGǥ on other backbones, I suspect their Gǣreally cheapGǥ Internet service will have to change.

If only the Internet was simple!
Mark Sullivan 12/5/2012 | 2:57:24 AM
re: Internet Peering on Thin Ice? Great post. Your take on the events of last week, and the events that led up to it, is certainly more sympathetic to Level 3 than the general tone of my article. I canGt completely deny your point about Cogent wanting to try the case with the "uninformed public," (via the media I suppose).

Still, regardless of who is really to blame, I think the larger point remains: the Internet can be disrupted because of the decisions of relatively few people, and for self-interested reasons, and consumers are left to pay the price. In this case, it was a bunch of enterprises who depend on the Internet to do their business.

In case this issue comes up again in the future, I would like to get your comments. If youGre up for that, you can shoot me an email at [email protected] -M
fiber_r_us 12/5/2012 | 2:57:24 AM
re: Internet Peering on Thin Ice? > Still, regardless of who is really to blame,
>I think the larger point remains:
> the Internet can be disrupted because of the
> decisions of relatively few people,
> and for self-interested reasons, and consumers
> are left to pay the price.
> In this case, it was a bunch of enterprises
> who depend on the Internet to do their business.

It has always been this way. Any business that is truly serious about connectivity to the Internet *must* buy connectivity from multiple providers to be able to survive this and other (more common) "outages".

Its very difficult for me to understand why a business that "depends on the Internet" would be so ignorant as to only buy service from a single provider. The Internet was only "disrupted" for these businesses due the "house of cards" these businesses built.

Uh, didn't LR get bit by an outage a few years ago due to being only homed to one provider?
sgamble 12/5/2012 | 2:57:22 AM
re: Internet Peering on Thin Ice? One thing not made clear by any posts or articles on this subjet that I have seen: Why were end-users blackholed from level3 or cogent? Does Cogent/Level3 only have PEERING and not TRANSIT with other providers? I am confused.

I run the BGP to my transit and peers. If a peer goes down my eBGP to my alternate Transit providers will kick in once the peer routes are withdrawn.

I have a connection to Cogent but by luck I had a prepend on Cogent and localpref setting that made it so reaching level3 went through another provider.

What's funny about my BGP settings to put traffic OFF of Cogent (I have transit with Cognet, Hydro One, Sprint/Rogers, Group Telecom...) is that I made these changes because I was seeing huge latency between Cogent and ATT one night. I called Cogent and after HOUNDING the ticket-jockey at the Cogent NOC he finally admitted they were in a billing dispute with AT&T! LR - dig! Get more information.

Sign of things to come folks. Level3 and ATT are going after them for more cash... Think Ill cancel my contract and get some MCI bandwidth ;)

pnni-1 12/5/2012 | 2:57:21 AM
re: Internet Peering on Thin Ice? Cogent will not be the only one to get slapped. I think allot of the Tier 2 providers who think they are Tier 1 will get dropped soon. Nothing is free, especially international transit links. Those that hold the SFI will be ok, or made to pay for the extra bandwidth. It has always worked that way.
sgamble 12/5/2012 | 2:57:21 AM
re: Internet Peering on Thin Ice? You're right I didn't understand that (always worked for the little guys apparently *grin*).

So Cogent and Level3 NEVER buy transit from the top ASNs like Sprint Net or MCI. Now I understand the dispute from people that the disruption can be huge when this happens. So really Cogent is thus viewed as a Tier2 provider now from both Level3 and AT&T. If this continues Cogent's $10/meg will have to go up or their model will fail. I am curious to know now what Tier1 providers charge Tier2 providers for transit /meg.

Funny I wouldn't have considered Level3 a Tier1 provider either only based on the amount of traffic destined to them from my network. With that said I only push about 400Mb at peak if I added all my transit pipes.

Thanks for the clarification.

fiber_r_us 12/5/2012 | 2:57:21 AM
re: Internet Peering on Thin Ice? sgamble, you mis-understand how the Internet works. As I mentioned in my earlier post, the Tier-1 backbone providers offer three basic types of connections:

1) You are a SFI peer. To become one of these you need to meet a few basic criteria:
-a) You have a nationwide backbone with peering capability in 10 or so major cities
-b) You are willing to pay for half of the connection cost between yourself and the SFI peer in each of those cities. These circuits are usually 2.5G or 10G in *each city*. So, setting this up is usually a multi-million dollar excercise for both parties due to the cost of router interfaces and "circuits" between the facilities.
-c) You have a customer base that creates a traffic pattern to and from your network that makes the traffic exchange more-or-less balanced between the two networks.
-d) Both parties agree that there is no point in trying to charge each other for bandwidth as neither party would substantially benefit.

SFI peering links only transport traffic between the two networks that is destined to those networks and those networks' customers. That is the BGP sessions only exchange routes belonging to the ASs of the two providers and those two provider's customers. These links are not used to reach anything else in the Internet.

2) Paid Peering. In this case you want to connect your network with another provider, but you don't meet conditions like those above. So you pay for the connectivity to the peer's network and pay for the traffic you send to that network. This type of peering's BGP works like number 1 above, but the traffic costs money.

3) Transit Peering. In this case you want to connect your network to another provider's network so that you can talk to:
-a) That provider
-b) That provider's customers, and
-c) The rest of the Internet (full transit), or some part of the rest of the Internet (partial transit).

For this last cast, the BGP sessions usually accept the "full" Internet route table (for full transit), or some agreed-upon partial table (for partial transit).

Smaller ISPs are usually only familiar with number 3.

Tier 1 backbones (Level 3, ATT, MCI, Sprint, etc) only do number 1 with each other and do not buy transit from anyone. That is, if you are a Tier 1 network, other networks connect to you either by:

A. SFI peering (other Tier 1 networks who are your *competitors*), or
B. paid peering or paid transit (non-tier 1 networks that are your *customers*)

For those networks that are neither A nor B above (such as yours), transit must be purchased through someone to get connectivity.

Thus, Level 3 has no transit, by design, to get to Cogent's network. They are not supposed to!

Cogent does some SFI peering with those that still let them (Level 3 now not wanting to be one of those), and buys partial transit for those not willing to be SFI peers with them (Sprint via transit from Verio for example).

Cogent's customers naturally purchase paid-peering or paid-transit from Cogent. You appear to be one of these.

However, Cogent did not arrange to have either paid peering or transit services established to the Level 3 network before the plug was pulled. It was their responsibility to do this, as have all others who have gone through similar de-peering. Since no agreement was in place for paid-peering directly to Level 3 or transit trough some third party, no one would send Level3 routes to Cogent. Anyone who did would carry all of that traffic on their backbone and naturally would want to be paid for it.

End users were not "blocked", Level 3 simply had no valid path to Cogent and vice-versa. If you were on one of the networks trying to get to the other, there simply was no route.

Fundamentally, to be an SFI peer, the relationship must be one where it wouldn't benefit either party to *try* to charge for bandwidth being exchanged. That is each party would essentially charge the other the same amount of money (thus going to a lot of trouble for no real income).

I suspect AT&T is similarly unhappy with Cogent's approach to the Internet and will also terminate their SFI with them unless things change.

I don't understand your last comment, as MCI follows the same basic logic and peering ideas. If MCI finds themselves in a position where a SFI peering agreement with someone is no longer beneficial, they too will pull the plug.

All of the major providers have done this before, including Cogent pulling the plug on others. It is the way peering is supposed to work.

I suspect that AT&T and MCI are a little careful right now due to their pending acquisitions. It might look bad if AT&T or MCI even *appeared* to be picking on some little guy right before their mergers (don't want anything to upset those big deals!).
slickmitzy 12/5/2012 | 2:57:20 AM
re: Internet Peering on Thin Ice? Hi All,

Fiber_r_us - thanks for your insight, it was very interesting.

I was wondering where does peering points like MAE, LINX and others get into the picture?

nemozen 12/5/2012 | 2:57:20 AM
re: Internet Peering on Thin Ice? Cogent had it coming. It is perfectly within Level 3's right to terminate peering, when it seizes to be in their interest.

Cogent is being disingenous and trying to manipulate the opinion of a misinformed public.
As recently as August 2005, Cogent reps were calling Level 3 customers offering unusually low prices with zero commit.
Also once Level 3 de-peered cogent, Cogent could have easily sent the traffic destined to Level 3 via their existing transit providers (Verio as mentionned in a previous post). But it would have cost them money, making their ultra-aggressive pricing unsustainable.

Some of us have been predicting this as long ago as mid 2002:
I remember at least one previous peering dispute involving Cogent, it was AOL in December 2002.

Indeed, given the way the Internet works today, it's only rational for Level 3 to de-peer Cogent. Clearly it's not worthwhile for Level 3 to exchange free with Cogent, when Cogent's aggressive pricing is shifting paying customers from Level 3 to Cogent. This is just legitimate defense, just as Cogent's aggressive pricing is legitimate offense.

Peering is a delicate equilibirum where two parties exchange traffic of roughly equal value. If the values start to diverge, tensions build up until they are released in a seismic event like this.

Now Cogent's cost for traffic from it's customer to Level 3's has jumped from $0 to something like $20/Mbps/month, more expensive than Cogent's price to its customers. So either they have to take that traffic at a loss, or just drop it and try to blame Level 3. Ultimately, Cogent's customers (who perhaps cannot be blamed for not being well informed enough) will pay a penalty for the overly agressive pricing they enjoyed to that point.

The real tragedy is if this were to prompt heavy-handed regulation from the FCC or international bodies.

The real solution would be efficient marketplaces where Cogent and Level 3 and everyone else would trade bandwidth and exchange traffic at real market prices. So when value is equal, it would be the same as settlement free peering, but when values diverge, that would be reflected in a gradual adjustment through the market, and money would change hands. Not with a sudden disruptive break as now, but with a dynamic market process.
(This last paragraph is a plug for my company invisiblehand.net)

Page 1 / 3   >   >>
Sign In