Controlling P2P TrafficControlling P2P Traffic

Peer-to-peer (P2P) IP networking goes to the very heart of the original Internet concept: distributed autonomous processes (peers) intercommunicating over a dumb packet network without any central control. But add the rapid and accelerating uptake of mass-market broadband IP access, and the result is rampaging bandwidth consumption that is threatening to choke the Internet on an uncontrolled diet of file-sharing traffic, much of it of dubious legality.

Welcome to the world of mass-market P2P media file sharing – the first killer application of the broadband age. It's both popular and bandwidth intensive. What more could an ISP want? Yet how much are they making out of it? Zilch, pretty much, because it is all part of an undifferentiated traffic stream (and mostly offnet, to boot). And what is it costing them? An arm and a leg as new capacity fills up, congestion grows, and P2P non-enthusiasts (they exist) complain more loudly about falling service quality as their traffic gets squeezed out.

On some networks, sophisticated measurements show that over 90 percent of traffic is P2P at times. Worse, ISPs may not even realize how much P2P traffic they have, as standard methods of analyzing traffic by looking at TCP ports are frequently fooled by P2P applications using random ports or those usually assigned to other applications.

But P2P doesn’t have to be the first ISP-killer application (the hyphen is important). P2P is actually a highly efficient and resilient method of distributing content over IP. So it makes a lot of sense from both an engineering and business viewpoint to think of ways of exploiting these virtues.

And it can be done, but only by making the network more intelligent and application-aware. Current network architectures just aren’t designed to cope with P2P traffic patterns or even to distinguish them. Yet, with the right tools and application management platforms, P2P traffic can be identified, classified, controlled – and eventually built into revenue generating services.

To find out how – and just how bad the P2P problem is becoming – read the following sections of this report. Here’s a hyperlinked list:

To view the Light Reading Webinar on which this report is base, click here.

There is useful background in the Light Reading White Paper archive: Broadband Network Solutions for Tiered/Premium Network Services & P2P Traffic Optimization - by P-Cube Inc.

And lots of links in Light Reading's recent coverage of this fast-moving subject:

— Geoff Bennett, Chief Technologist, Heavy Reading

P2P is evolving rapidly, and already three distinct generations of technology have emerged.

In the beginning was Napster, which was a centralized system, so the file-sharing database was held on Napster-owned servers. This proved to be a legal weakness and was a key factor in the court’s ruling to shut down the company because it allowed the charge of contributory infringement. That is, Napster knew that its users were engaged in copyright infringement, and Napster also provided the site and the facilities for users to do so.

The second generation was typified by the first version of Gnutella, which distributed everything (files and file-location database) to avoid the legal issues of Napster. So all nodes in this kind of network are peers. But this type of extreme distribution leads to performance problems, because Gnutella, in trying to locate a particular file, would send out several queries to a group of peers, which might then have to send more queries to further peers, and so on. So just locating a server with the required file can generate a lot of network traffic, nearly all of which is wasted. This is a very inefficient use of network resources.

The third generation is a hybrid of the first two; current examples are New Gnutella, FastTrack, KaZaA, and Grokster. In KaZaA, for example, some of the nodes become supernodes, which hold the lists of files and the locations of the clients from which they can be downloaded. The term "client" here is in the context of the control plane of KaZaA, where ordinary nodes only have control connections to supernodes, and the supernodes form their own higher-level mesh of connections.

It’s Terabytes Now

P2P applications are the most downloaded applications on the network, and P-Cube Inc. reckons that P2P now accounts for 50 to 70 percent of all Internet traffic. KaZaA alone has more than 230 million client downloads, with about 900 million files available for sharing, representing about 6,500 terabytes.

Different clients have become popular in different parts of the world. Although KaZaA leads worldwide, Winny leads in Japan, with runners-up eDonkey and WinMX. Newcomers on the scene include Piolet and BitTorrent.

So in network terms P2P is big – and it is creating big problems for carriers and service providers.

P2P seems to have a nearly infinite appetite for bandwidth, which threatens to exhaust a provider’s existing network resources. So if a service provider doesn’t upgrade its network capacity to meet this demand, its customers will be likely to switch to another provider that does. But basic broadband access is already becoming a commodity, which means the average revenue per user (ARPU) is falling. This commoditization is caused by the explosion of choice, at least in North America, which then leads to price wars because the service is best-effort only, and price becomes the sole differentiator.

So P2P seems to trap service providers between falling revenues on one side and rising capacity investment costs on the other – effectively giving an ugly negative return on investment. Provider survival is going to depend on abuse control and the meaningful offering of tiered levels of service for different types of application. To do this, providers must look much more closely at their network usage and how it can be managed.

Peer-to-peer applications have some very striking – and somewhat unexpected – effects on aggregate network traffic flows. Critically important is that upstream and downstream flows are pretty much symmetrical. In other words, the aggregate volume of traffic downstream over a period is similar to the volume sent upstream.

This matters, because mass-market broadband access networks (such as residential DSL) are designed on the basis that downstream traffic will greatly dominate – as it does in standard applications such as Web browsing and streaming media.

Specifically, ADSL (asymmetric DSL) technologies are generally configured to have about three times the bandwidth coming down to the user as going up to the network. The whole assumption of ADSL is that it can steal some of the upstream spectrum without damaging application performance. But if flows are symmetrical, this argument flies out the window.

At the level of a single subscriber doing a P2P download, the traffic is, of course, asymmetric because there is a small customer file sent upstream, followed by a larger file being transferred downstream. So the download traffic by itself looks just like the client/server model of the Web. But, because subscribers are sharing files as peers, at any given time a home PC may be a client or a server, so this combination of different subscribers uploading and downloading at the same time makes the aggregate traffic for file sharing symmetrical.

Figures 1 and 2 show how such symmetry occurs in measurements made on a real network. Upstream P2P traffic in Figure 1 fluctuates between about 50 and 100 Mbit/s during the month studied; downstream P2P traffic in Figure 2 for the same network and period fluctuates between about 30 and 70 Mbit/s.

The Figures also make plain the absolute dominance of P2P over other forms of application traffic – between 50 percent and 70 percent of upstream bandwidth, for example, and peaking at 20 times the volumes of the second largest application (Web HTTP).

“We have actually seen some of the upstream peer-to-peer traffic take over 90 percent of the total bandwidth upstream in some of the service providers’ networks,” says Kurt Dobbins, chief technology officer of Ellacoya Networks Inc. “One of the interesting aspects is that this traffic-bandwidth phenomenon didn’t exist three years ago, nor two years ago, nor when most of these access networks were designed.”

And P2P traffic is always on. Figure 1 shows this, as it is basically where the troughs are on the red trace. The sustained P2P file-sharing rate in this sample is about 60 Mbit/s, right through peak and offpeak hours. So a lot of file sharing occurs in the background – probably even when some subscribers are sleeping, having left their PCs switched on.

Another critical fact is that bandwidth for standard interactive applications is most needed during peak hours, as this is when most customers are active and form their perceptions of service quality. To have interactive bandwidth and Web traffic competing with huge quantities of P2P traffic during these hours is a recipe for trouble.

So P2P users are creating a vast wall of bytes that is hitting the Internet. Ellacoya Networks measured 37 terabytes in total over 35 days behind one 12,000-subscriber cable-modem termination system (CMTS) alone (see Figure 3), utterly dwarfing the other application categories.

“We have seen this across cable operators, service providers, and universities. These graphs and charts, even if they are from a particular trial, represent traffic that is experienced anywhere,” says Ellacoya’s Dobbins.

Yet, as Figure 4 shows, these P2P users can account for only about a quarter of online users at a time (although this proportion is increasing) – yet they can be hogging over 90 percent of the bandwidth. In comparison, the majority of the subscribers (65 percent) was using the Web, but consuming only 7 percent of the bandwidth; and just over 50 percent of subscribers were using instant messaging (IM), consuming only 1 percent of the total bandwidth. Note that some users are using multiple applications, so these percentages do not total 100 percent.

This pattern is very stable and recurs day after day.

The upshot is that the popularity of P2P traffic is causing network-capacity issues for providers, not only because of total amount of traffic, but also from its unique nature compared to traditional applications, such as browsing and email. Table 1 summarizes how P2P differs from traditional applications and the problems these differences cause.

Table 1: Peer-to-Peer Compared to Traditional Applications

Says Milind Gadekar, vice president for marketing at P-Cube: “P2P was not a planned application from the service-provider viewpoint, and now they are actively looking at a solution to try to address these issues.”

However, this isn’t easy, because P2P applications have some pretty weird characteristics compared to the applications and protocols service providers are used to dealing with. They tend to be nonstandard or proprietary, and are often developed by hackers with the deliberate intention of fighting back against efforts made to control them. So they can use port hops to random port numbers, including those assigned to other applications, and some are now moving to using an encrypted mode (for example, Winny, which uses random port selection as well).

“It becomes very difficult for the existing transport infrastructure – whether a router, a CMTS, or a B-RAS [broadband remote-access server] – to identify these peer-to-peer applications and control or manage them,” says Gadekar. “What’s required to control and manage the peer-to-peer applications is the ability to do the accurate classification of the traffic, either through deep packet analysis or by looking for particular signatures.”

So what can service providers do to combat the problems caused by P2P? Quite a lot, as it happens. There are many different possible solutions, and the good news is that these can be deployed together in certain combinations.

Option one is just to keep cranking up the bandwidth. This is fine to a very limited extent, but P2P tends to expand to fill the bandwidth available and drives a lot of off-net bandwidth, which is expensive for service providers. And they don’t get any extra revenue from their customers who are running P2P traffic.

So, why not just shut down P2P? There are several reasons why this isn’t going to happen. Service providers could potentially be sued by the P2P companies, for instance. But probably the biggest reason is that the first ISP that blocks KaZaA will lose a huge number of customers overnight; these are the applications that are driving the rollout of residential broadband. However, there’s a special case with the enterprise network. Here the CIO really must get a grip over nonbusiness P2P traffic and shut it down. CIOs have a solid legal position, and the Recording Industry Association of America (RIAA) has threatened to sue companies that allow their networks to be used for the downloading of copyright-infringing files.

So some ISPs have tried to impose usage caps to change user behavior. A user exceeding a certain cumulative traffic volume during a month can be shut down, or the traffic is given a low priority, or the user may incur additional bandwidth charges. In practice, such caps go down like a lead balloon with customers. In the U.K., NTL Inc. (Nasdaq Europe: NTLI) had only to mention that it was thinking of applying bandwidth caps to receive a wave of complaints from users.

A fourth option is to prioritize traffic to throttle back P2P volumes. However, this can be hard with conventional IP filters because P2P traffic tends to masquerade as other traffic.Another option is to cache content, as shown in Figure 5, where two local networks at the bottom are linked by an IXC that provides the backbone. There are three steps to caching.

In step 1, Joe pulls down a new movie from, say, Ann, who is on a different local network. This is off-net traffic, but, as it goes to Joe, a copy is held in the cache in the bottom left network. This caching forms step 2, although it occurs in parallel to step 1.

Step 3 takes place when another user in Joe’s local network, say, Fred, asks for the same movie from Ann. Instead of the file coming over the IXC network, this time it’s pulled down from the local-network cache, with a big saving in off-net traffic.

Of course, caching makes sense only if the frequency distribution of downloaded files is heavily skewed, but this seems to be so. It turns out that a big chunk of P2P traffic comes from a tiny proportion of files – there are definitely top-100 lists for both music and movie files.

The issue with caching may well be its legality. ISPs are currently safe from prosecution by the RIAA because they aren’t responsible for the nature of traffic that passes through their network. However, if they use caches, they are technically storing illegal material. The legal position has yet to be settled.

The final option is the most elegant: apply traffic management and service control to the P2P traffic. This puts control back in the hands of the service provider, which can now create tiered services or apply priorities as desired.

Service Control

Service control and optimizing the network for peer-to-peer traffic is a two-step process. The first step is to gain a much better, or more granular, view of the traffic on the network.

According to P-Cube’s Gadekar, service providers can be wildly wrong in their view of their network traffic.

“At one service provider where our equipment has been deployed, the provider had been using some of the router logs to analyze the traffic on the network,” he says. “Based on these statistics, they saw that close to 80 percent of the traffic was on port 80, and they assumed it was Web traffic. After they deployed our solution, we were able to identify that close to 60 percent of the overall traffic was being generated by KaZaA Lite, which masquerades, or uses, port 80 as a way of preventing providers from clearly identifying it and controlling it.”

The goal is to develop a clearer view of what applications are running on the network, and to analyze this information to understand traffic patterns – for example, by time of day, or whether traffic is off-net or on-net traffic – and even to identify some of the top users of P2P.

The second step is to control the amount of P2P traffic on the network, taking into account the other applications that are running and the service provider’s overall objectives for the network. So the provider might decide to halve the bandwidth currently being used by P2P and instead use the freed-up capacity to improve the service experience for other applications – and also to introduce new revenue-generating services.

“In one of our deployments the provider decided to cut the peer-to-peer traffic by close to 50 percent,” says Gadekar. “The call volume of subscribers complaining about overall service experience then dropped to zero. They did get some calls from people complaining about the peer-to-peer usage, but typically subscribers are much happier with the overall service experience.”

In particular, upstream file-sharing bandwidth control is very effective at reducing bandwidth and goes undetected by an ISP’s paying subscribers, as typically over 93 percent of traffic is served off-net to nonsubscribers.

Service providers have a lot of choices in deciding how and where to control P2P bandwidth. They can do it on an aggregate or on a per-protocol basis, or control upstream only, or control downstream only. They can apply control in both directions and at different rates. And control can be applied to an entire group of subscribers, or just to subgroups or even individual subscribers. And all these options can be combined in various ways, and applied differently at different times during the day.

To give a flavor of how these approaches can pan out, Figure 6 shows how an operator experimented with P2P rate controls only on upstream traffic and its aggregate. The point of controlling only P2P upstream traffic is that it affects only the rate at which files are being shared, not their download rate. So an individual subscriber retains the expected broadband download speeds and service, but the reduced file-sharing rate is typically not obvious.

In Figure 6 the provider chose in step 2 to allocate a DS3’s worth of bandwidth to P2P file sharing upstream. All other non-P2P applications continue to run unencumbered from rate controls.

According to Ellacoya’s Dobbins, a provider would typically begin by bringing P2P traffic down to its original sustained rate. Then the provider would progressively reduce it down to an acceptable multiple of other application traffic.

“It is really not about taking bandwidth away – it’s about providing fairness and improving service quality on the network,” he says. “What is interesting is how quickly P2P traffic can consume bandwidth. By removing rate control, the experiment shows how P2P applications consume the new bandwidth within hours.”

Figure 7 shows another refinement to service control – time-shifting P2P traffic into offpeak periods by applying rate limiting during peak hours. As soon as rate limiting is lifted, bandwidth is immediately consumed. The effect is to time-shift the P2P bandwidth out of the peak hours (light blue) and into the offpeak hours (dark blue).

But there is a further angle to service control and traffic management that potentially goes far beyond P2P congestion control and capacity optimization: differentiated services.

By implementing service control in the network, service providers are adding a layer of intelligence to make the network smarter. Being able to identify P2P applications and optimize the network clears the ground for the next stage of providing differentiated services.

They can create new application-based plans, which is a new way of thinking for them. Traditionally, providers have concentrated on speed tiers through, for example, Gold, Silver, and Bronze services offering decreasing bandwidths. But application-based plans could involve tiers such as:

Further, providers can think in terms of driving the consumption of legal content by providing new services – for example, by faster downloads and integrated billing with content distribution partners.

“What that P2P traffic is really showing is this incredible, almost insatiable, appetite for digital content,” says Ellacoya’s Dobbins. “Broadband access networks are really becoming content-distribution networks, so we think that there is a new service opportunity for content owners, aggregators, and distributors to use the broadband access network for distribution of copyright-protected works”.

Apple iTunes is a specific example of this opportunity. Each download of a $0.99 iTunes track is treated as a separate transaction and attracts a standard credit-card fee to Apple of 1.735% + $0.20 (about $0.22). Financially, this is not very efficient, both for Apple and the content providers. Apple could, of course, improve matters by grouping transactions, but service providers historically have been masters at billing efficiently for small amounts. So collecting payments efficiently for content providers could become an important new revenue stream for the service providers.

But to make this transformation possible, providers need a new piece of kit: a service-management platform. This has to perform four tasks:

Once service-management platforms are installed, service providers can transform their broadband networks into efficient, intelligent, application-aware distribution networks. And this presents a real opportunity to turn P2P into a legitimate application to everyone’s benefit.

P2P is undoubtedly the killer application that broadband has been waiting for, but so far the commercial opportunity has been missing. The people who are effectively funding the P2P revolution at the moment are unfortunately the copyright owners. But one very positive thing that has come out of P2P is that it is a very efficient and very resilient distribution-network architecture.

So there is a real opportunity here for broadband service providers to take P2P as a serious business proposition for content distribution. And there is growing evidence – witness iTunes – that subscribers are willing to pay for legal content if it is reasonably priced and easy to obtain. Providers would receive carriage fees for the content they carry, while offering faster downloads as a premium channel. By integrating download transactions onto a cable or broadband bill they could eliminate transaction fees – one of the bugbears of distribution. And they would be exploiting the high subscriber penetration of Tier 1 operators.