Light Reading
Packet loss is plaguing Project Lightspeed's IPTV service, say sources – and a few unhappy IPTV subscribers

AT&T Still Has IPTV 'Jitters'

Light Reading
News Analysis
Light Reading
8/11/2006
50%
50%

Industry sources say AT&T Inc. (NYSE: T) is struggling with video packet loss on the eve of the major market launch of its U-verse IPTV service.

Word has it the U-verse network loses roughly two packets of data per minute. “A lost video packet is more than 1,400 bytes of information, and that's going to cost you a half second of video,” one source says. For the viewing public that can mean little annoyances like screen pixelation and jitter -- or, at worst, full screen freezes.

In the U-verse distribution network, video packets hop from AT&T’s video super headend, to regional headends, to the local central offices, to nodes in the neighborhoods. At each "hop," packets can arrive in incorrect order or overload the buffers within the routers and switches, leading to losses.

An AT&T spokesman chose not to comment on the packet loss issue.

Microsoft Corp. (Nasdaq: MSFT), which supplies the IPTV middleware for the U-verse network, is said to be hard at work increasing its product's capacity for dealing with lost packets. Using a software algorithm called Resilient UDP, the set-top box, upon detecting missing or misplaced packets, sends a "resend" command up through the network. The missing packets are then sent down from a server. (See Microsoft Soups Up the Set-Top.)

That works fine if traffic levels are normal, the source says. But if packet loss should occur during the final minutes of the Super Bowl, a million resend requests could pummel the network at the same time, seriously burdening the system.

One source close to the situation says Microsoft has already built in a 15 to 30 second delay to live video streams to allow some time for dealing with packet loss. AT&T, the source says, is uneasy about the scaleability of the setup.

Microsoft TV Edition product manager Jim Baldwin says his company's middleware platform adds roughly a quarter of a second delay for packet error correction and another second of delay for instant channel changing, but that's it. Baldwin says AT&T is perfectly happy with his company’s Resilient UDP approach to packet loss, but says AT&T may decide to use forward error correction as well.

The Scientific-Atlanta Inc. encoders at the headend of the system, Baldwin says, will add a certain amount of latency to the video streams, but AT&T will decide how much. Scientific-Atlanta chose not to comment for this story. (See AT&T, Verizon Tout Telco TV .)

Meanwhile, AT&T may be hedging its bets. Sources close to the situation say AT&T engineers are experimenting with forward error correction (FEC) on one leg of AT&T's video network. Applied near the headend encoders, FEC adds additional video bitstreams that can be used to reconstruct damaged streams on the fly, downstream in the network. Also at the headend, the video packets are tagged sequentially so that the system can detect interruptions in the packet order downstream. The FEC technology monitors the bitstreams at several points in the network, including the set-top box, to detect missing or damaged packets.

Baldwin says FEC works well when the type and size of the packet loss is predictable. But IP networks can lose a whole packet, groups of consecutive packets, or just one bit within a packet, he says.

U-verse was already under criticism, as analysts have noted the initial Project Lightspeed rollout of broadband access won't have the punch to carry high-definition TV, and some question whether Lightspeed will roll out as quickly as AT&T hopes. (See Is Lightspeed Slowing?)

Flickers of trouble?
U-verse debuted in late June in AT&T’s home turf of San Antonio, Texas, and the AT&T spokesman says the service is getting good reviews from users. The few actual U-verse users contacted by Light Reading say the packet loss issue, at least so far, hasn’t been very visible on their TVs. (See AT&T to Launch Lightspeed Next Month.)

“The flicker [pixelation] has only appeared a few times,” says U-verse user Alan Weinkrantz. “But keep in mind that I also have HD Cable from TimeWarner, and there are times when I also get flicker or a nano-of-a-second blur on that.” Weinkrantz, whose day job is as a technology public relations man, has become a minor celebrity through his U-verse user's blog.

“I had some pixelation the very first day after I got service,” writes another U-verse customer, Chad Brantly. “The next day, I got a call from AT&T. They said that they had been having some problems in my area, but they had just done a hardware update and things should be better. Since then, I haven't seen any pixelation. The service has been great,” Brantly writes.

A few unhappy U-verse adopters, however, have shown up on a bulletin board called Uverseusers.com. One of the five discussion threads on the site is called “pixelations and freeze-ups,” wherein three users -– “eapinsatx,” “dilbert” and “nohbdy” -– complain of moderate to serious pixelation and screen freezes.

Heavy Reading analyst Rick Thompson points out the San Antonio debut is probably happening in a very "controlled" network environment. With the technology world watching, AT&T is surely taking steps to make sure the fledgling service makes a good first impression.

But the issue of scaleability looms for AT&T’s engineers, as the carrier plans to roll out U-verse in 15 to 20 markets by the end of 2006. (See AT&T Readies Lightspeed in North Texas.) "It's one thing to get a complex technology like IPTV rolled out to a group of a few thousand users," says independent telecom analyst Kermit Ross. "But it’s quite another thing to kick that up a notch to a few hundred thousand and then yet another thing to kick it up to a few million users."

In other words, once AT&T cranks up the numbers, any failure to tame packet loss will be evident on millions of TV screens.

— Mark Sullivan, Reporter, Light Reading

(30)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View
Page 1 / 3   >   >>
trzwuip
50%
50%
trzwuip,
User Rank: Light Beer
12/5/2012 | 3:44:58 AM
re: AT&T Still Has IPTV 'Jitters'
does this mean the d-servers dont work the way microsoft promised?
metroman
50%
50%
metroman,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
There is another possibility.

If the traffic is being delivered using multicast, the network elements will need to replicate the traffic. Depending upon the hardware in use, it may be that they are having issues either with the replication itself or the forwarding mechanisms for multicast. Ingress replication for multicast may create congestion in switch fabrics, while if the stream is being replicated to many egress interfaces, all interfaces need to be available before the frame is forwarded. In this case it only take one congested interface to cause network-wide packet loss and service degradation.

The network elements need to have robust hardware to deal with multicast replication and well thought out prioritisation & buffering mechanisms.

Does anyone know if they have the same issues on VoD (unicast)?

Metroman
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'

My guess is that the small buffers on the STB to have the short channel switch time is the issue. Pretty much any network imperfections flow through to the STB. Scaling this will be pretty funny, especially that there is no HDTV.

seven
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
I realy don't understand that there are that much packets lost per minute. Packet loss is more or less dependable on congestion and buffer depths, and if the article is true then there is a lot of congestion on the path from Master head end to the Customer. In my opinion a case of bad network planning/packet prioritization (I expect that this video traffic has a higher priority than Internet traffic..) that can be solved fairly easy. Or is there something I don't see ???

stolsma
alcatell1
50%
50%
alcatell1,
User Rank: Light Beer
12/5/2012 | 3:44:56 AM
re: AT&T Still Has IPTV 'Jitters'
or possibly fatal flaws in 7450 multicast throughput
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'

Couple of things. You are all making an assumption that any packet loss is due to forwarding issues, prioritization, or congestion.

1 - 1 Packet (potentially 1 Bit Error) in 2 minutes of data transmission is a BER of around 10-9 for an SDTV stream (1 error in approximately 240 Mb). Ever checked the BER of a DSL line?

2 - The buffers are very small on MS STBs to achieve the fast channel change. In normal IPTV setups, there are seconds of buffering and channel switches look like Satellite systems. Network jitter must be minimized to keep these STBs fed. This limits abilities of the multicast network to buffer in replication - as such buffering will add jitter as more paths are added to that channel.

seven
gianconstantine
50%
50%
gianconstantine,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
I think you may not be considering the nature of multicast vs. unicast in delivering video. The bandwidth requirements on the backplane are not heavy enough to cause problems. Consider 250 channels encoded in VC1. While high, let's consider 3Mbps stream rate. Even if all 250 channels are being subscribed to off all cards in 10 blade chassis (for argument's sake), the switching fabric would be pushing less than 8Gbps.

I ran an IPTV network in Atlanta for all of 2005. I slammed Cisco 6500 Sup720/MSFC3 with multicast and unicast traffic and the switch fabric stats were as clean as a whistle.

Microsoft and Alcatel have some sort of funky setup that's screwing them. This problem is not so simple as an incapable switch or router.

Gian.
cabecar
50%
50%
cabecar,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
VoD should not have the same issues as it is not Multicast traffic. The problem with VoD is that each new instance requires a new stream. This means that they need to have Video pump engines close to the edge... if they don't a spike in VoD usage can affect users over the entire network. If they do have pump engines near the edge then a spike will only affect edge users in a particular region. To prevent this they need to implement video CAC for Unicast VoD.
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
Thanks for all the reply's. It's all a little bit clearer to me now..

I know something about the 7450 and I'm almost sure there is nothing wrong with the multicast replication (as sure you can be after 9 months of testing..). And to be honest we didn't see any alarming packet loss (except some bit errors on bad optical lines that resulted in bad packets, resolved by replacing optical cards, can happen any time).

What we did see was packet loss on DSL lines because of impuls noise and bad copper lines (comparable to seven's 10-9), btw some impuls ratio settings and DSL CRC checking can help there. Trade off is the added latency for other traffic like gaming or internet pings. When I wrote my first message I was still under the impression that Lightspeed was FttH but I saw that also FttC (with DSL as last/first mile) was used.

Personally I think that solving this problem with unicast retransmit requests is not the way to go but who am I...

stolsma
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
cabecar wrote: ".... To prevent this they need to implement video CAC for Unicast VoD."

Anyone out there who can clarify if AT&T is doing something like this ??? I know some operators who are trying to implement a function this. Some with great results!!

stolsma
Page 1 / 3   >   >>
Educational Resources
sponsor supplied content
Educational Resources Archive
Flash Poll
From The Founder
It's clear to me that the communications industry is divided into two types of people, and only one is living in the real world.
LRTV Interviews
CenturyLink: Building the Case for NFV

12|19|14   |   02:14   |   (0) comments


At the 2020 Vision Executive Summit, James Feger, VP, Network Strategy & Development at CenturyLink, talks about how the US operator is approaching virtual network functions from an operational and business case perspective.
LRTV Interviews
Liberty Global Sees Business Goldmine

12|18|14   |     |   (0) comments


Steen Sorensen, VP of business services for Liberty Global, explains where the giant international MSO sees growth potential.
LRTV Documentaries
EE: The Road to 5G

12|16|14   |   16:02   |   (1) comment


Andy Sutton, the principal network architect at UK mobile operator EE, explains how his company is using Wembley stadium as a wireless test bed and how that's helping EE to plan the evolution to 5G.
LRTV Huawei Video Resource Center
Highlights of Huawei's NFV Open Cloud Forum 2014

12|16|14   |     |   (0) comments


Huawei hosted its inaugural NFV Open Cloud Forum during the SDN & OpenFlow World Congress 2014 in Düsseldorf, Germany. The Forum brought together technology thought leaders, senior executives and telecom professionals from global carriers, industry associations, as well as other partner companies in the ecosystem, to exchange views on and collectively explore how ...
LRTV Custom TV
Realizing Operators' Digital Vision

12|16|14   |   5:23   |   (0) comments


Leveraging technology is fundamental to digital transformation but understanding customers and serving them really well is at the heart of digital businesses. TM Forum lists the following as the strategic pillars of the digital business: business agility and rapid innovation, operational agility and effectiveness, IT and data centricity, plus customer centricity. ...
LRTV Documentaries
US Cellular Injects Analytics Into LTE

12|16|14   |   2:57   |   (1) comment


US Cellular's Mario Vela explains how the operator uses analytics for network planning and what comes next as the carrier looks to eke more value out of its metrics.
LRTV Interviews
How Cox Biz Plans to Keep Growing

12|15|14   |     |   (2) comments


Steve Rowley, SVP of Cox Business, details how the third-biggest US MSO intends to boost its revenues to $2 billion and beyond over the rest of the decade
LRTV Huawei Video Resource Center
Interview With Bill Zhang, Director of SoftCOM Product Management, Huawei

12|15|14   |   2:50   |   (0) comments


Bill Zhang elaborated on Huawei's open philosophy in NFV solution development and network architecture design at the SDN & OpenFlow World Congress 2014.
LRTV Huawei Video Resource Center
Event Highlights: Huawei at SDN & OpenFlow World Congress 2014

12|15|14   |   3:43   |   (0) comments


Huawei joined the 2014 SDN & OpenFlow Congress as one of the key sponsors and contributors. At the event, Huawei reinforced the openness and flexibility of its network infrastructure strategies, and provided updates on its SDN and NVF innovations. Through participations at the exhibitions, forums and speeches, Huawei encouraged the industry to "think bigger and ...
LRTV Interviews
How Cable Biz Services Hit $10B Mark

12|12|14   |     |   (1) comment


Cable operators reached $10 billion in annual business services revenues by delving deeper into their vertical markets and expanding beyond the smallest firms.
LRTV Documentaries
Mediacom Aims to Test Connected Tractors

12|11|14   |   05:07   |   (3) comments


Cable business service provider is taking its services to the 'agribusiness' sector in partnership with farm equipment specialist John Deere and is getting involved in Gigabit Cities developments.
LRTV Interviews
TWC Business Looks Beyond $3B

12|10|14   |     |   (0) comments


TWC Business Services chief Phil Meeks explains how his unit has reached $3 billion in annual revenues and what its plans are for next year.
Upcoming Live Events
February 10, 2015, The Westin Peachtree Plaza, Atlanta, GA
March 17, 2015, The Cable Center, Denver, CO
April 14, 2015, The Westin Times Square, New York City, NY
May 6, 2015, McCormick Convention Center, Chicago, IL
May 13-14, 2015, The Westin Peachtree, Atlanta, GA
June 9-10, 2015, Chicago, IL
Hot Topics
T-Mobile, BlackBerry Flirt With Reuniting
Sarah Reedy, Senior Editor, 12/17/2014
1-Gig: Coming to a Small Town Near You
Jason Meyers, Senior Editor, Gigabit Cities/IoT, 12/17/2014
Comcast Launches 4K Streaming Service
Alan Breznick, Cable/Video Practice Leader, 12/18/2014
US Carriers, You're Going to Cuba!
Mitch Wagner, West Coast Bureau Chief, Light Reading, 12/18/2014
T-Mobile Lights Up 27 Wideband LTE Cities
Sarah Reedy, Senior Editor, 12/15/2014
Like Us on Facebook
Twitter Feed
Webinar Archive