Light Reading
Packet loss is plaguing Project Lightspeed's IPTV service, say sources – and a few unhappy IPTV subscribers

AT&T Still Has IPTV 'Jitters'

Light Reading
News Analysis
Light Reading
8/11/2006
50%
50%

Industry sources say AT&T Inc. (NYSE: T) is struggling with video packet loss on the eve of the major market launch of its U-verse IPTV service.

Word has it the U-verse network loses roughly two packets of data per minute. “A lost video packet is more than 1,400 bytes of information, and that's going to cost you a half second of video,” one source says. For the viewing public that can mean little annoyances like screen pixelation and jitter -- or, at worst, full screen freezes.

In the U-verse distribution network, video packets hop from AT&T’s video super headend, to regional headends, to the local central offices, to nodes in the neighborhoods. At each "hop," packets can arrive in incorrect order or overload the buffers within the routers and switches, leading to losses.

An AT&T spokesman chose not to comment on the packet loss issue.

Microsoft Corp. (Nasdaq: MSFT), which supplies the IPTV middleware for the U-verse network, is said to be hard at work increasing its product's capacity for dealing with lost packets. Using a software algorithm called Resilient UDP, the set-top box, upon detecting missing or misplaced packets, sends a "resend" command up through the network. The missing packets are then sent down from a server. (See Microsoft Soups Up the Set-Top.)

That works fine if traffic levels are normal, the source says. But if packet loss should occur during the final minutes of the Super Bowl, a million resend requests could pummel the network at the same time, seriously burdening the system.

One source close to the situation says Microsoft has already built in a 15 to 30 second delay to live video streams to allow some time for dealing with packet loss. AT&T, the source says, is uneasy about the scaleability of the setup.

Microsoft TV Edition product manager Jim Baldwin says his company's middleware platform adds roughly a quarter of a second delay for packet error correction and another second of delay for instant channel changing, but that's it. Baldwin says AT&T is perfectly happy with his company’s Resilient UDP approach to packet loss, but says AT&T may decide to use forward error correction as well.

The Scientific-Atlanta Inc. encoders at the headend of the system, Baldwin says, will add a certain amount of latency to the video streams, but AT&T will decide how much. Scientific-Atlanta chose not to comment for this story. (See AT&T, Verizon Tout Telco TV .)

Meanwhile, AT&T may be hedging its bets. Sources close to the situation say AT&T engineers are experimenting with forward error correction (FEC) on one leg of AT&T's video network. Applied near the headend encoders, FEC adds additional video bitstreams that can be used to reconstruct damaged streams on the fly, downstream in the network. Also at the headend, the video packets are tagged sequentially so that the system can detect interruptions in the packet order downstream. The FEC technology monitors the bitstreams at several points in the network, including the set-top box, to detect missing or damaged packets.

Baldwin says FEC works well when the type and size of the packet loss is predictable. But IP networks can lose a whole packet, groups of consecutive packets, or just one bit within a packet, he says.

U-verse was already under criticism, as analysts have noted the initial Project Lightspeed rollout of broadband access won't have the punch to carry high-definition TV, and some question whether Lightspeed will roll out as quickly as AT&T hopes. (See Is Lightspeed Slowing?)

Flickers of trouble?
U-verse debuted in late June in AT&T’s home turf of San Antonio, Texas, and the AT&T spokesman says the service is getting good reviews from users. The few actual U-verse users contacted by Light Reading say the packet loss issue, at least so far, hasn’t been very visible on their TVs. (See AT&T to Launch Lightspeed Next Month.)

“The flicker [pixelation] has only appeared a few times,” says U-verse user Alan Weinkrantz. “But keep in mind that I also have HD Cable from TimeWarner, and there are times when I also get flicker or a nano-of-a-second blur on that.” Weinkrantz, whose day job is as a technology public relations man, has become a minor celebrity through his U-verse user's blog.

“I had some pixelation the very first day after I got service,” writes another U-verse customer, Chad Brantly. “The next day, I got a call from AT&T. They said that they had been having some problems in my area, but they had just done a hardware update and things should be better. Since then, I haven't seen any pixelation. The service has been great,” Brantly writes.

A few unhappy U-verse adopters, however, have shown up on a bulletin board called Uverseusers.com. One of the five discussion threads on the site is called “pixelations and freeze-ups,” wherein three users -– “eapinsatx,” “dilbert” and “nohbdy” -– complain of moderate to serious pixelation and screen freezes.

Heavy Reading analyst Rick Thompson points out the San Antonio debut is probably happening in a very "controlled" network environment. With the technology world watching, AT&T is surely taking steps to make sure the fledgling service makes a good first impression.

But the issue of scaleability looms for AT&T’s engineers, as the carrier plans to roll out U-verse in 15 to 20 markets by the end of 2006. (See AT&T Readies Lightspeed in North Texas.) "It's one thing to get a complex technology like IPTV rolled out to a group of a few thousand users," says independent telecom analyst Kermit Ross. "But it’s quite another thing to kick that up a notch to a few hundred thousand and then yet another thing to kick it up to a few million users."

In other words, once AT&T cranks up the numbers, any failure to tame packet loss will be evident on millions of TV screens.

— Mark Sullivan, Reporter, Light Reading

(30)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View
Page 1 / 3   >   >>
trzwuip
50%
50%
trzwuip,
User Rank: Light Beer
12/5/2012 | 3:44:58 AM
re: AT&T Still Has IPTV 'Jitters'
does this mean the d-servers dont work the way microsoft promised?
metroman
50%
50%
metroman,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
There is another possibility.

If the traffic is being delivered using multicast, the network elements will need to replicate the traffic. Depending upon the hardware in use, it may be that they are having issues either with the replication itself or the forwarding mechanisms for multicast. Ingress replication for multicast may create congestion in switch fabrics, while if the stream is being replicated to many egress interfaces, all interfaces need to be available before the frame is forwarded. In this case it only take one congested interface to cause network-wide packet loss and service degradation.

The network elements need to have robust hardware to deal with multicast replication and well thought out prioritisation & buffering mechanisms.

Does anyone know if they have the same issues on VoD (unicast)?

Metroman
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'

My guess is that the small buffers on the STB to have the short channel switch time is the issue. Pretty much any network imperfections flow through to the STB. Scaling this will be pretty funny, especially that there is no HDTV.

seven
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
I realy don't understand that there are that much packets lost per minute. Packet loss is more or less dependable on congestion and buffer depths, and if the article is true then there is a lot of congestion on the path from Master head end to the Customer. In my opinion a case of bad network planning/packet prioritization (I expect that this video traffic has a higher priority than Internet traffic..) that can be solved fairly easy. Or is there something I don't see ???

stolsma
alcatell1
50%
50%
alcatell1,
User Rank: Light Beer
12/5/2012 | 3:44:56 AM
re: AT&T Still Has IPTV 'Jitters'
or possibly fatal flaws in 7450 multicast throughput
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'

Couple of things. You are all making an assumption that any packet loss is due to forwarding issues, prioritization, or congestion.

1 - 1 Packet (potentially 1 Bit Error) in 2 minutes of data transmission is a BER of around 10-9 for an SDTV stream (1 error in approximately 240 Mb). Ever checked the BER of a DSL line?

2 - The buffers are very small on MS STBs to achieve the fast channel change. In normal IPTV setups, there are seconds of buffering and channel switches look like Satellite systems. Network jitter must be minimized to keep these STBs fed. This limits abilities of the multicast network to buffer in replication - as such buffering will add jitter as more paths are added to that channel.

seven
gianconstantine
50%
50%
gianconstantine,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
I think you may not be considering the nature of multicast vs. unicast in delivering video. The bandwidth requirements on the backplane are not heavy enough to cause problems. Consider 250 channels encoded in VC1. While high, let's consider 3Mbps stream rate. Even if all 250 channels are being subscribed to off all cards in 10 blade chassis (for argument's sake), the switching fabric would be pushing less than 8Gbps.

I ran an IPTV network in Atlanta for all of 2005. I slammed Cisco 6500 Sup720/MSFC3 with multicast and unicast traffic and the switch fabric stats were as clean as a whistle.

Microsoft and Alcatel have some sort of funky setup that's screwing them. This problem is not so simple as an incapable switch or router.

Gian.
cabecar
50%
50%
cabecar,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
VoD should not have the same issues as it is not Multicast traffic. The problem with VoD is that each new instance requires a new stream. This means that they need to have Video pump engines close to the edge... if they don't a spike in VoD usage can affect users over the entire network. If they do have pump engines near the edge then a spike will only affect edge users in a particular region. To prevent this they need to implement video CAC for Unicast VoD.
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
Thanks for all the reply's. It's all a little bit clearer to me now..

I know something about the 7450 and I'm almost sure there is nothing wrong with the multicast replication (as sure you can be after 9 months of testing..). And to be honest we didn't see any alarming packet loss (except some bit errors on bad optical lines that resulted in bad packets, resolved by replacing optical cards, can happen any time).

What we did see was packet loss on DSL lines because of impuls noise and bad copper lines (comparable to seven's 10-9), btw some impuls ratio settings and DSL CRC checking can help there. Trade off is the added latency for other traffic like gaming or internet pings. When I wrote my first message I was still under the impression that Lightspeed was FttH but I saw that also FttC (with DSL as last/first mile) was used.

Personally I think that solving this problem with unicast retransmit requests is not the way to go but who am I...

stolsma
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
cabecar wrote: ".... To prevent this they need to implement video CAC for Unicast VoD."

Anyone out there who can clarify if AT&T is doing something like this ??? I know some operators who are trying to implement a function this. Some with great results!!

stolsma
Page 1 / 3   >   >>
Educational Resources
sponsor supplied content
Educational Resources Archive
Flash Poll
LRTV Custom TV
VeEX – Live from the Show

8|21|14   |   5:58   |   (0) comments


An overview of VeEX Test and Measurement solutions including TX300S multi-service test set with VeExpress cloud-based management system, UX400 universal modular platform supporting 100G testing, and the redesigned RXT modular platform.
LRTV Custom TV
Transitioning CE 2.0 Networks Into the SDN & NFV Era With Telco Systems

8|19|14   |   5:19   |   (0) comments


Telco Systems' Ariel Efrati (CEO) and Moshe Shimon (VP of Product Management) discuss virtualization and how the company's new Open Metro Edge solution utilizes the SDN and NFV concepts to accelerate and orchestrate service delivery through its innovative product portfolio and software applications.
LRTV Custom TV
NFV Myths: Is NFV Still Several Years Away?

8|11|14   |   1:13   |   (0) comments


Some say that NFV (network functions virtualization) is still several years away from being implemented on mobile operator networks. This isn't the case. Operators can get started on their paths to NFV now, as this short video from Skyfire shows.
LRTV Custom TV
A New Security Paradigm in SDN/NFV

7|28|14   |   02:54   |   (0) comments


Paul Shaneck, Global Director Network Solutions for Symantec, discusses the evolving virtualized network, explaining how Symantec is leading the security discussion as it relates to SDN and NFV, and helping to ensure the network is protected and compliant.
LRTV Documentaries
Sprint's Network Evolution

7|24|14   |   14:59   |   (0) comments


Sprint's Jay Bluhm gives a keynote speech at the Big Telecom Event (BTE) about Sprint's network and services evolution strategy, including Spark.
LRTV Documentaries
BTE Keynote: The Software-Defined Operator

7|24|14   |   18:43   |   (1) comment


Deutsche Telekom's Axel Clauberg explains the concept of the software-defined operator to the Big Telecom Event (BTE) crowd.
Light Reedy
Numbers Are In: LR's 2014 Salary Survey

7|24|14   |   1:25   |   (7) comments


Our fourth annual Salary Survey paints a picture of who's hiring, firing, earning, and yearning for a change in the telecom industry.
LRTV Custom TV
Driving the Network Transformation

7|23|14   |   4:29   |   (0) comments


Intel's Sandra Rivera discusses network transformation and how Intel technologies, programs, and standards body efforts have helped the industry migration to SDN and NFV.
LRTV Custom TV
Distributed NFV-Based Business Services by RAD

7|18|14   |   5:38   |   (0) comments


With the ETSI-approved Distributed NFV PoC running in the background, RAD's CEO, Dror Bin, talks about why D-NFV makes compelling sense for service providers, and about the dollars and cents RAD is putting behind D-NFV.
LRTV Custom TV
MRV Accelerating Packet Optical Convergence

7|15|14   |   6:06   |   (0) comments


Giving you network insight to make your network smarter.
LRTV Custom TV
NFV-Enabled Ethernet for Generating New Revenues

7|15|14   |   5:49   |   (0) comments


Cyan's Planet Orchestrate allows service providers and their end-customers to activate software-based capabilities such as firewalls and encryption on top of existing Ethernet services in just minutes.
LRTV Custom TV
Symkloud NVF-Ready Video Transcoding, Big Data

7|9|14   |   3:41   |   (0) comments


Kontron and ISV partner Vantrix demonstrate high-performance video transcoding and data analytic solutions on same 2U standard platform that is ready for SDN and NFV deployments made by mobile, cable and cloud operators.
Upcoming Live Events!!
September 16, 2014, Santa Clara, CA
September 16, 2014, Santa Clara, CA
September 23, 2014, Denver, CO
October 29, 2014, New York City
November 6, 2014, Santa Clara
November 11, 2014, Atlanta, GA
December 9-10, 2014, Reykjavik, Iceland
June 9-10, 2015, Chicago, IL
Infographics
Today's Cartoon
Hot Topics
Level 3 Does Big Data Differently
Carol Wilson, Editor-at-large, 8/21/2014
IoT Alert: Samsung Snaps Up SmartThings
Mari Silbey, Independent Technology Editor, 8/15/2014
Comcast Streams Back to School
Mari Silbey, Independent Technology Editor, 8/21/2014
Sprint Drops Prices, But Also Speeds?
Sarah Reedy, Senior Editor, 8/21/2014
Line-Powered Phone Lines: A Hot Topic Again
Carol Wilson, Editor-at-large, 8/20/2014
Like Us on Facebook
Twitter Feed