Netflix's Problem Is Its Transit Network – Report

A veteran engineer with decades of Internet experience is challenging Netflix's claims that ISPs are slowing the performance of its streaming video service, attributing the problem instead to insufficient middle-mile bandwidth from the transit companies Netflix uses to connect its content delivery network to last-mile networks. Also in play are other factors, including choices Netflix users can control.

Peter Sevcik, president of engineering consultant NetForecast Inc. , has been engaged in network design and performance analysis since the early days of the Internet, when he was part of the team at BBN Technologies, whose work in packet-switching led to the founding of the Internet. Today he runs a consulting company that advises ISPs, and also is engaged in the SamKnows project in support of the Federal Communications Commission (FCC) 's Measuring Broadband America project.

His conclusions are largely based on analyzing the way Netflix Inc. (Nasdaq: NFLX) delivers its streaming video service, and on the Netflix ISP Speed Index , which the video company itself created to assess how different ISPs are delivering its streaming video service. The details of Sevcik's analysis can be found in his report, "How the Netflix ISP Speed Index Documents Netflix Speed Problems." (See Netflix Hearts Google Fiber and YouTube Joins Netflix, Ranks ISPs.)

Using Netflix numbers
Sevcik analyzed the Netflix ISP Speed Index over a nine-month period, from September 2013 to May 2014, that includes the four-month stretch during which Netflix says its service delivery was degraded over major networks including Comcast Corp. (Nasdaq: CMCSA, CMCSK) and Verizon Communications Inc. (NYSE: VZ). He looked specifically at the 12 ISPs whose speed ratings deteriorated during that time period.

There are two key pieces to Sevcik's analysis: First, because of how Netflix uses adaptive streaming and compression in its streaming video service, a quality viewing experience of a Netflix video only takes about 2 Mbit/s in the last mile, and most ISPs are delivering much more bandwidth than that to their end users.

"It never needs to go that fast. That is one of beauties of NetFlix's adaptive streaming and compression process," he says. There is a bump at the beginning of a video download to fill the viewing buffer, but from that point forward, the video doesn't need more than 2 Mbit/s to stream, and the second the viewer stops watching, that stops as well.

Secondly, the Netflix ISP Speed Index shows that performance of all 12 of the ISPs improved following the point at which Comcast and Netflix directly connected the latter's video servers to the cable giant's last mile networks, thereby removing the Comcast traffic from the middle-mile connections provided by Cogent Communications Holdings Inc. (Nasdaq: CCOI) and Level 3 Communications Inc. (NYSE: LVLT). That change is reflected in the chart below, drawn by NetForecast from the Netflix numbers.

Source: NetForcecast
Source: NetForcecast

"In October to December, all of the curves are going down, showing negative change," Sevcik points out. "Netflix was slowing down -- the average per-user throughput that was being delivered via Netflix to those players kept going down all the way to January, which seems to be the low point, and stays there for February."

At the end of February, when the Comcast-Netflix arrangement kicks in, everyone's delivery performance improves, and six of the 12 get even better than they were in September of 2013, he says. "Removing about one quarter of the traffic that was destined to Comcast off the middle-mile network gave it the relief to go ahead and fill up all that interconnection capacity."

All of these ISPs were connected to the Netflix CDN via the video company's choice of middle-mile providers -- Level 3 and Cogent. Netflix and Cogent have said the issue is that the local ISPs refuse to upgrade the equipment at those interconnection points to handle the high volume of Netflix traffic. (See Cogent Gearing for Another Peering Battle and Verizon Threatens to Sue Netflix.)

A business decision?
If that were the primary problem, Sevcik says, then those networks would show continued degradation of service, and not be affected by the movement of the traffic headed for Comcast subscribers off the middle-mile transit networks. Sevcik doesn't fault the middle-milers -- but he points out Netflix could choose at any time to buy more middle-mile capacity and probably address that issue.

Instead, Sevcik points out, Netflix is pushing its Open Connect strategy, which embeds Netflix CDN servers directly into the data centers of broadband ISPs. And now, with Comcast, it has opted for a direct connection.

"Netflix has a problem," he says. Its traffic represents about one-third of the Internet traffic in-use at any given point, and distributing all that traffic is a challenge. The company seems to be moving away from the middle-mile strategy to pushing its content closer to consumers via the Open Connect approach, and possibly more direct connection deals to come with the biggest players.

Other factors affecting how well Netflix videos perform are actually under the control of end users, even though they may not understand that. Sevcik believes the Netflix ISP Speed Index also reflects the ability of the end user's device to process highly encoded video, as well as menu selections users can make for higher-quality video.

"Many Netflix users don't know the menu option exists," he says. "They don't realize they can go to a 'Manage Video Quality' menu and choose from good, better or best. But even if you are opting for the best performance, if you are using an old PC or any device that can't process highly encoded video, then you are going to get a [standard definition] video at best."

Light Reading contacted Netflix to ask for a response to Sevcik's claims. In an email reply, the video company restated its belief that interconnection points with broadband ISPs are the source of streaming problems:

    Congestion occurs at the ISP interconnection point for only a few broadband providers because they fail to provide the Internet access their customers pay for. Like any traffic jam, if three lanes onto a bridge are closed, cars will back up onto the connecting highway. The middle mile isn't experiencing congestion for many ISPs, who either increase capacity to meet consumer demand, or work cooperatively with Netflix to bring our video closer to mutual subscribers.

There are ISPs that rank highly on the Netflix ISP Speed Index without directly connecting to Netflix, the company stated.

— Carol Wilson, Editor-at-Large, Light Reading

Page 1 / 4   >   >>
Joe Stanganelli 7/14/2014 | 12:34:23 AM
Next step I'm just waiting for Netflix to start its own ISP and make the whole thing moot.

Except I don't trust Reed Hastings to be the CEO to handle that kind of job.

So I think we're stuck with what we have for the time being.  :p
Carol Wilson 7/10/2014 | 9:23:52 PM
Re: Chart Victor, 

A lot of what you are saying makes sense. The main thing again is it leads away from blaming the local ISPs for deliberately slowing down traffic, which is what Netflix are doing. 

brookseven - the equpment of which you speak, is it the interconnection gear that Netflix has been complaining about or metro networking gear that is gumming up the works, in your opinion?
VictorRBlake 7/10/2014 | 7:15:58 PM
Chart Two additional comments (not about the article but) about the chart. One -- what that data shows is that the performance across alll ISPs is highly correlated with FEW exceptions (BHN take off when others are headed down and the direct peering -- which obviously always improves performance). When the performance goes down on one -- it goes down and all, and when it goes up on one, it goes up on nearly all of them. Again, high correlation across all ISPs. What this points to is "what's the same across all of them" and that is .... Netflix's network and systems obviously.

The second thing the data shows is that performance dropped going into the winter and rose coming out of winter returning to spring. Putting on my rocket science hat here, it seems pretty clearly -- and this is well known in the video services industry -- that folks watch more video when it is COLD out and less when the weather improves. If you understand that then it's obvious -- there are huge performance problems with Netflix. When their load is heavy performance suffers, when the load is light, performance is better. Again -- points back to Netflix.

IF -- and the data does not show this, the performance on one ISP stood out as different, then you'd have something to look at. Did Netflix peer directly with BHN when the performance went up before the dotted line ? Why ? What's the explanation if it isn't peering ? I don't know the answer, but that's what Netflix engineers should be asking.

IMHO, this has nothing to do with the FCC. This is old fashioned Internet engineering. Otherwise known to network engineers as work. It's way to do -- look at the data - identify not only the problems, but what is performing well and look to copy what is good and fix what is bad. I don't see how the FCC is going to help that process.
sam masud 7/10/2014 | 11:26:12 AM
Re: Why middle mile? To say that Netflix doesn't care about user experience--now that's profound. I guess Netflix is tired of making money and is in the mood for suppuku. Okay, I'll buy that...
brookseven 7/10/2014 | 11:02:50 AM
Re: More peering transparency needed? Carol, I agree that the local loop is not the problem. It never has been. That is very different than saying the local ISP. There are all kinds of metro equipment in the way. seven
Carol Wilson 7/10/2014 | 10:55:23 AM
Re: More peering transparency needed? I completely agree that greater transparency around peering agreements is necessary. I suspect most carriers don't want that information public because it would impact their negotiations and because as long as it remains cloaked in secrecy it is always easy to blame someone else. 

Where I think Peter Sevcik's comments are interesting and worth considering is that he is saying based on his engineering experience that the local loop isn't the culprit which most folks are saying it is. I think it is always a good idea to consider contrarian views. 
Carol Wilson 7/10/2014 | 10:53:36 AM
Re: do your homework -- OR READ MORE CLOSELY I'm not backing down from the story - or my emphasis - because it accurately reflects my interview of Peter Sevcik. What I am saying is that, as a journalist, I have a responsibiity to ask the questions but I can't always indepenently verify the information. His point is that it is the Netflix' compression and adaptive streaming technique that make this possible, and I immediately confess not to be an expert at either.

My decision to run with the story is based on Peter Sevcik's credibility and I think that is well-established. You may not agree with his conclusions, but they are accurately reflected in the article. 
mhhf1ve 7/9/2014 | 9:38:44 PM
Re: More peering transparency needed? gconnery, I believe you are correct and the FCC is looking into making peering agreements more transparent (at least to the FCC) -- the obstacle is that the service providers are objecting and saying that revealing such proprietary statistics would hurt them competitively in the market.  
gconnery 7/9/2014 | 8:01:15 PM
Re: do your homework -- OR READ MORE CLOSELY Carol,

Its your story, either you stand by it or you don't.  You are the one summarizing the report, and you are the one who chose to start the story with the "you only need 2Mbps" emphasis.  I'm not sure where the 2Mbps number comes from, its neither the lowest rate that Netflix can stream at, nor the highest as others point out.  A simple google search for "netflix streaming Mbps" turns up the previously mentioned:


which I would have expected you to turn up as well.  Minimum speed to stream Netflix?  0.5Mbps.  Max for HD?  5Mbps.  I'm not sure where the 2Mbps number comes from myself. 

I think the report makes a reasonable case without this 2Mbps stuff.  And you are the one who chose to emphasize it. 
gconnery 7/9/2014 | 7:55:38 PM
Re: Why middle mile? You can't imagine why Netflix might want to degrade their customers experience?

Okay, look I'm generally on your side and all but let's not say stupid things.  Netflix gets your money every month whether they do a good job delivering your service or not.  Sure if its bad enough you might drop them, but they might also walk a fine line where you get so-so service, they convince you to blame somebody else for the problem, and they spend less in network fees. 

I'm not saying this is what is happening, I'm saying you can't say stuff like that.
Page 1 / 4   >   >>
Sign In