Level 3 interrupted a free, or "settlement free," peering connection with Cogent last Wednesday, which resulted in tens of millions of IP addresses effectively vanishing for many enterprises using either Level 3 or Cogent for Internet access.
How could that happen? Hunter Newby of telx Group Inc. says part of the answer is the way the Internet is set up. He notes that the Internet comprises ISP (and other non-commercial) networks that are loosely stitched together on handshake deals among a very select group known as "peering managers."
"IP peering is an interesting and delicate business,” Newby says. “It is made up of people and relationships that are rarely documented and have no remedy provisions, no real definition of breach, no damages provisions, etc."
In other words, it's business done on a handshake, with little legal documentation. That makes it easier to clip a peering connection, even if consumers will be negatively affected.
“How L3 peering managers can make a decision like that and just de-peer a network as large as Cogent is amazing,” Newby observes. He characterizes Level 3’s action as one of the biggest disruptions in the history of the Internet.
Of course, Level 3 doesn’t see it quite like that. “I completely take exception to that characterization,” says Level 3 COO Kevin O’Hara, adding that his company has de-peered many other networks in the past with “little or no” disruption in Internet service. (See EarthLink Uses Level 3.)
“Our goal all along has been to make sure no consumer or user of the Internet is impacted by the commercial issues happening deep within the bowels of our network,” he says.
But there could be a lesson here: Enterprises might want to investigate whether their ISPs have contingency plans in place so that access to the Internet is never put in jeopardy.
Who is really to blame for the events last week is now mostly lost in a fog of claims and counterclaims. Level 3 says Cogent has been sending more traffic over Level 3's network than they've been sending in the other direction, and that they should therefore pay a tariff. Cogent, meanwhile, insists that it has satisfied all of the various traffic volume and bandwidth metrics in Level 3’s peering requirements, and should be allowed to continue peering with Level 3 at no cost.
Cogent CEO Dave Schaeffer told Light Reading Wednesday that the real reason Level 3 clipped the peering connection last week is because it believes Cogent is pricing its Internet service at “below cost,” making it hard for Level 3 to compete.
"In mid-September I got a phone call from a very senior person at Level 3, and in that telephone call they said, 'We’re going to cut you off if you guys don’t either pay us or reconsider your pricing strategy and raise prices,' ” Schaeffer says. (See Cogent: King of Ports .)
Cogent sells Internet access for around $10 per megabyte, according to the company, while the Level 3 rate is somewhere between $25 and $60 per megabyte. (See Cogent Revenues Jump in Q2.)
Level 3’s O’Hara strongly denies Schaeffer’s claim. “I have heard this allegation before and we take it very seriously, and to my knowledge nobody from this organization has made that request of Cogent, and that’s as far as I’m going to go with that,” O’Hara told Light Reading Thursday.
O’Hara says after Cogent didn’t respond to several notifications that de-peering was imminent, his company assumed Cogent had made a comparable peering arrangement elsewhere.
When it became apparent that Cogent had not, and that many users were effectively cut off from large portions of the Internet, Level 3 reversed itself. Two days after the de-peering, the connection between the two networks was restored, and Level 3 says it will remain so until November 9.
In the meantime, Cogent must either strike a deal with Level 3 or find a comparable peering arrangement elsewhere. As of Thursday, the two companies hadn't talked.
Historically, the U.S. Congress and Federal Communications Commission (FCC) have left it to the network operators to work out their peering arrangements. But some worry that if peering disputes between IP networks become more common, Congress and the FCC may be compelled to act.
“When Congress sees something like this happening and it looks as if consumers are the ones suffering, they naturally want to jump in and solve the problem,” says Staci Pies of the Voice On the Net (VON) Coalition. “The question is, whether jumping in and solving the Level 3/Cogent problem, if in fact there is a problem that needs to be solved, does damage elsewhere.
“We are very much in favor of Net neutrality principles,” Pies adds, “But we are not in favor of interconnection obligations and standards for IP-to-IP networks.”
Former FCC chief of staff and current Legg Mason Inc.
analyst Blair Levin agrees with Pies that it would take far more than just one peering dispute to push Congress or the FCC to act. But Levin also points out that the current atmosphere of consolidation and convergence in IT and telecom may increase the risk of such disagreements in the future.
If anybody stayed awake through that last post, I'm thinking I should discuss "de-peering" as well as it pertains to the generality of the term "peering".
The term "de-peering" does not, IMHO, only refer to SFI but applies equally (though less commonly) to partial and to private peering scenarios as well.
We've discussed a situation where one party withdrew from a SFI (and withdrew routing advertisements) and the term "de-peering" applied.
But if the players are in a private peering situation and one of them pulls the plug (maybe the other wasn't paying the bill), the term "de-peering" can still be applied.
If you don't have a redundant PoP with your peer (private or no), and the building collapses, you're just as disconnected ... although because there was no commercial intent to do so, we might prefer to avoid the term "de-peering" for this one.
nemozen, #20, "Peering is competitive, yes, but that does not mean it's not on thin ice. In fact peering (I mean SFI) is an unstable equilibrium state, because there's a discontinuity between peering (SFI) and the alternatives (paid peering or transit)."
What you say is strictly true. But I don't find it in disagreement with what I said. We're simply overloading the term "peering" in slightly different ways, and I think that's leading to a confusing conclusion.
Let's imagine a peering point (we'll call it NAP1) with, say, 10 local ISPs (named ISP10 thorugh ISP19) all having a presence and all agreeing to deliver traffic to their subscribers for each of the other networks at no charge. They are peering (under SFI), but among all of them there's no reachability to the Greater Internet Beyond.
Now let's say that one of those 10 has grown (say, ISP10), and now has a presence at NAP2 in another city, with a similar makeup at NAP2.
But ISP10 doesn't want to become a transit network for everybody's traffic, just because it's now at another NAP. So it advertises reachability of its own subscribers to all the other ISPs at each of NAP1 and NAP2.
As ISP10 continues to grow, it enters a commercial agreement with ISP21 (at NAP2) to carry ISP21's traffic to NAP1 and to any of the ISPs there. So it advertises reachability of ISP[10..19] to ISP21, but to nobody else at NAP2. (Return traffic isn't covered.) ISP10 and ISP21 are still peering, but are no longer under a settlement-free interconnect (SFI). Some may use the term partial peering, or, if they connect separately from other ISPs at the POP, private peering.
Eventually, ISP10 grows to the point where it splits itself internally into an ISP and an NSP (Network Service Provider), with the NSP picking up the commercial transit traffic part of the business and the ISP looking after such things as flaky DSL modem connections, email, etc for their subscribers.
And NSPs can repeat the cycle, agreeing to carry each others' traffic with no settlement fees involved, or charging for the privilege as commercial needs dictate.
And I think it's at the point where the distinction between an ISP and an NSP is blurred, and again at the point where NSPs bring vastly different volumes of traffic (and networking value) to the table that you find the "unstable equilibrium" for peering.
You pay someone for access to their network in order to achieve reachability to everyone else their network can reach.
All that SFI really means is that the business managers have decided that they know they'd be billing each other comparable amounts each month, so they'll choose to forego the accounting costs instead.
I once talked with a fellow who was surprised to learn that his IP traffic went (at the time) from Oakland to San Francisco by way of Chicago, simply because that's how the peering arrangements worked it out. (He was really upset to learn that his DSL was using ATM on the carrier.)
So the commercial arrangements may become shaky at times, just like anything else in the economy, but peering itself is competitive. (Unless, as you say, you only mean SFI when you use the term "peering" on its own. But then the conversation becomes really difficult to follow.)
From the links in my earlier post, one can find a list of NAPs which describe themselves as "neutral". Presumably, that suggests some sort of SFI among the participants. But in all likelihood, they're also banding together to buy common access to the Internet Beyond - they have a private peering arrangement in place (or two or more), but they all pay for it and access it as a consortium rather than individually.
> I hope that folks will understand that Internet > Peering is competitive, even if the details can > become rather complex, and isn't on any thinner > ice than other parts of the economy.
Peering is competitive, yes, but that does not mean it's not on thin ice. In fact peering (I mean SFI) is an unstable equilibrium state, because there's a discontinuity between peering (SFI) and the alternatives (paid peering or transit).
It's hard for a relationship to evolve from peering to non-peering without a huge disruption on the cost side. When Cogent gets de-peered, the price of the same Mbps to the same destination on L3 jumps from $0 to the transit price of the provider that Cogent could use to reach L3 (*).
Once the traffic ratios, or the business relationships, or any aspect of the peering justifications change, it becomes like two tectonic plates pushing against each other. The tension builds and builds, until it is released in an earthquake, with unpredictable damaging effects all around.
Cogent got caught with their pants down because the prices they charge their customers are artificially low, and locked in with flat-rate contracts, so Cogent cannot cover their true costs which include a certain probability of having to pay for transit to Level 3.
Now if both sides were interconnecting on a real market-price basis, they would always have a price at which they can exchange traffic. Sometimes it would be equal value in both directions (as in SFI), so the net flow of money would be $0. But, other times, the value in one direction would be more (either because there's more traffic or because its a higher value network), and the market prices would adjust to reflect that fact. http://www.invisiblehand.net
-nemo-
(*) I mean the IP bandwidth cost -- of course there are other costs e.g. ports, circuits, etc. which are roughly unchanged between peering and transit.
While I've appreciated the explanations which have been given, I thought a couple other comments might broaden the subject a little bit more, using more colloquial and less formal terms.
First, the concept of a "stub" AS (Autonomous System), at least as far as Internet Routing is concerned. This would be a network which does not provide any transit services for other networks. It might be a corporate network, or it could be a regional ISP (even one that provides Internet access for some corporate customers which themselves might operate "stub" AS's ... but let's not get confused by the reality).
Then, we have all heard of the concept of "hot potato" routing: that a network will hand off traffic to the next network in the chain towards the destination AS at the earliest opportunity.
Peering agreements, of course, are what governs what that "earliest opportunity" actually is: they limit the advertisement of what ASs you can reach through any particular other network.
Even at the same Network Access Point (NAP, or Point of Presence (PoP); MCI's are known as MAEs - they say it's not an acronym although it supposedly originally came from "Metropolitan Area Ethernet" back when they were regional peering points ... but I digress) different carriers with a presence might have different commercial arrangements, different interconnect rules (google: Routing Arbiter), and therefore very different BGP tables and different reachability. It's not just a "free-for-all" of interconnections.
There was a time when MCI (mae.net), Merit (merit.edu), NANOG (nanog.org), and ISI (e.g. http://www.isi.edu/div7/) had much more useful information available online; however, I can only find a shadow of it now. And I've already demonstrated to those in the know just how long I've been away from the current details.
Since Google brought it up, I'll also refer to sbcbackbone.net and to wikipedia.org/wiki/PAIX (which tells me I have to re-learn some more acronyms which have changed meaning ... sigh).
I hope that folks will understand that Internet Peering is competitive, even if the details can become rather complex, and isn't on any thinner ice than other parts of the economy.
Because their business model was to be the "low cost" provider to "business customers" (i.e. servers: connections that *generate* lots of traffic, but don't *consume* much). The idea being that you can sell Internet connections at $10/Mb/mo if your infrastructure only included this specific, easy-to-serve customer base.
However, this ignores the reality that paid peering/transit to other providers costs at least twice that much! So, Cogent's model only works if you can convince other providers to carry your *server* traffic to the *consumers* on the other provider's networks at little or no cost to Cogent.
Other providers actually have been carrying Cogents traffic for free since the bubble. Then, in the last few years, AOL, Sprint, a few others, and now Level3 have called bullsh*t on the whole plan and refused to continue to carry Cogent's traffic for free. Go figure, I'm amazed that Cogent has got away with it this long.
No provider carriers "all" routes. In fact, it would be difficult to define what "all" even really means. The Internet is not completely "contiguous" from a routing perspective. There are always holes that are not reachable for various reasons.
Now, in the case of Level 3 not being able to reach Cogent, this is Cogent's problem for not arranging either paid peering with Level 3 or transit through Verio (or some other third party) for connectivity to Level 3.
Level 3's Tier 1 status is derived from their network customer base and their SFI interconnects with other Tier 1s. Connectivity between Level 3 and Cogent (a non-Tier 1) is Cogent's responsibility and has nothing to do with Level 3's "status".
Why should Level 3 (or any other provider) be responsible for Cogent not wanting to pay for traffic that Cogent wants to dump on them in order to maintain their cheap rates??? Either Cogent *pays* for the traffic, or they don't get connected.
How can L3 be Tier one if they don't carry Cogent's routes ? Regardless of how they get them L3 can hardly claim to be Tier I if they don't carry them.
Also, the IXs can not really support the volume of traffic that go between even two of the big players. I hadn't looked recently and don't know the exact numbers, but the majors (Sprint, Level3, AT&T, MCI, etc) SFI peer independedntly with each other in about 10 locations around the country with each location being a 10G connection.
That is, Level 3 has: ~ten 10G connections to AT&T, *AND* has ~ten 10G connections to MCI, *AND* has ~ten 10G connections to Sprint, etc...
Thus, it is possible that 10s (or even 100s) of Gigabits/sec of traffic can flow between two major backbones.
The big backbones are also connected at the "public peering" points, primarily for connections to the dozens (hundreds?) of smaller ISPs who simply don't require a lot of peering bandwidth in a lot of locations.
As Tony mentioned, peering at these Internet Exchanges (IXs) can be any of the forms of peering that I described before (SFI, paid peer, paid transit, etc). Each company makes its agreement with each other company independently.
The blogs and comments are the opinions only of the writers and do not reflect the views of Light Reading. They are no substitute for your own research and should not be relied upon for trading or any other purpose.
To save this item to your list of favorite Light Reading content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Digg
Del.icio.us
Reddit
