Light Reading

AT&T Still Has IPTV 'Jitters'

Light Reading
News Analysis
Light Reading
8/11/2006
50%
50%

Industry sources say AT&T Inc. (NYSE: T) is struggling with video packet loss on the eve of the major market launch of its U-verse IPTV service.

Word has it the U-verse network loses roughly two packets of data per minute. “A lost video packet is more than 1,400 bytes of information, and that's going to cost you a half second of video,” one source says. For the viewing public that can mean little annoyances like screen pixelation and jitter -- or, at worst, full screen freezes.

In the U-verse distribution network, video packets hop from AT&T’s video super headend, to regional headends, to the local central offices, to nodes in the neighborhoods. At each "hop," packets can arrive in incorrect order or overload the buffers within the routers and switches, leading to losses.

An AT&T spokesman chose not to comment on the packet loss issue.

Microsoft Corp. (Nasdaq: MSFT), which supplies the IPTV middleware for the U-verse network, is said to be hard at work increasing its product's capacity for dealing with lost packets. Using a software algorithm called Resilient UDP, the set-top box, upon detecting missing or misplaced packets, sends a "resend" command up through the network. The missing packets are then sent down from a server. (See Microsoft Soups Up the Set-Top.)

That works fine if traffic levels are normal, the source says. But if packet loss should occur during the final minutes of the Super Bowl, a million resend requests could pummel the network at the same time, seriously burdening the system.

One source close to the situation says Microsoft has already built in a 15 to 30 second delay to live video streams to allow some time for dealing with packet loss. AT&T, the source says, is uneasy about the scaleability of the setup.

Microsoft TV Edition product manager Jim Baldwin says his company's middleware platform adds roughly a quarter of a second delay for packet error correction and another second of delay for instant channel changing, but that's it. Baldwin says AT&T is perfectly happy with his company’s Resilient UDP approach to packet loss, but says AT&T may decide to use forward error correction as well.

The Scientific-Atlanta Inc. encoders at the headend of the system, Baldwin says, will add a certain amount of latency to the video streams, but AT&T will decide how much. Scientific-Atlanta chose not to comment for this story. (See AT&T, Verizon Tout Telco TV .)

Meanwhile, AT&T may be hedging its bets. Sources close to the situation say AT&T engineers are experimenting with forward error correction (FEC) on one leg of AT&T's video network. Applied near the headend encoders, FEC adds additional video bitstreams that can be used to reconstruct damaged streams on the fly, downstream in the network. Also at the headend, the video packets are tagged sequentially so that the system can detect interruptions in the packet order downstream. The FEC technology monitors the bitstreams at several points in the network, including the set-top box, to detect missing or damaged packets.

Baldwin says FEC works well when the type and size of the packet loss is predictable. But IP networks can lose a whole packet, groups of consecutive packets, or just one bit within a packet, he says.

U-verse was already under criticism, as analysts have noted the initial Project Lightspeed rollout of broadband access won't have the punch to carry high-definition TV, and some question whether Lightspeed will roll out as quickly as AT&T hopes. (See Is Lightspeed Slowing?)

Flickers of trouble?
U-verse debuted in late June in AT&T’s home turf of San Antonio, Texas, and the AT&T spokesman says the service is getting good reviews from users. The few actual U-verse users contacted by Light Reading say the packet loss issue, at least so far, hasn’t been very visible on their TVs. (See AT&T to Launch Lightspeed Next Month.)

“The flicker [pixelation] has only appeared a few times,” says U-verse user Alan Weinkrantz. “But keep in mind that I also have HD Cable from TimeWarner, and there are times when I also get flicker or a nano-of-a-second blur on that.” Weinkrantz, whose day job is as a technology public relations man, has become a minor celebrity through his U-verse user's blog.

“I had some pixelation the very first day after I got service,” writes another U-verse customer, Chad Brantly. “The next day, I got a call from AT&T. They said that they had been having some problems in my area, but they had just done a hardware update and things should be better. Since then, I haven't seen any pixelation. The service has been great,” Brantly writes.

A few unhappy U-verse adopters, however, have shown up on a bulletin board called Uverseusers.com. One of the five discussion threads on the site is called “pixelations and freeze-ups,” wherein three users -– “eapinsatx,” “dilbert” and “nohbdy” -– complain of moderate to serious pixelation and screen freezes.

Heavy Reading analyst Rick Thompson points out the San Antonio debut is probably happening in a very "controlled" network environment. With the technology world watching, AT&T is surely taking steps to make sure the fledgling service makes a good first impression.

But the issue of scaleability looms for AT&T’s engineers, as the carrier plans to roll out U-verse in 15 to 20 markets by the end of 2006. (See AT&T Readies Lightspeed in North Texas.) "It's one thing to get a complex technology like IPTV rolled out to a group of a few thousand users," says independent telecom analyst Kermit Ross. "But it’s quite another thing to kick that up a notch to a few hundred thousand and then yet another thing to kick it up to a few million users."

In other words, once AT&T cranks up the numbers, any failure to tame packet loss will be evident on millions of TV screens.

— Mark Sullivan, Reporter, Light Reading

(30)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View        ADD A COMMENT
Page 1 / 3   >   >>
trzwuip
50%
50%
trzwuip,
User Rank: Light Beer
12/5/2012 | 3:44:58 AM
re: AT&T Still Has IPTV 'Jitters'
does this mean the d-servers dont work the way microsoft promised?
metroman
50%
50%
metroman,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
There is another possibility.

If the traffic is being delivered using multicast, the network elements will need to replicate the traffic. Depending upon the hardware in use, it may be that they are having issues either with the replication itself or the forwarding mechanisms for multicast. Ingress replication for multicast may create congestion in switch fabrics, while if the stream is being replicated to many egress interfaces, all interfaces need to be available before the frame is forwarded. In this case it only take one congested interface to cause network-wide packet loss and service degradation.

The network elements need to have robust hardware to deal with multicast replication and well thought out prioritisation & buffering mechanisms.

Does anyone know if they have the same issues on VoD (unicast)?

Metroman
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'

My guess is that the small buffers on the STB to have the short channel switch time is the issue. Pretty much any network imperfections flow through to the STB. Scaling this will be pretty funny, especially that there is no HDTV.

seven
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
I realy don't understand that there are that much packets lost per minute. Packet loss is more or less dependable on congestion and buffer depths, and if the article is true then there is a lot of congestion on the path from Master head end to the Customer. In my opinion a case of bad network planning/packet prioritization (I expect that this video traffic has a higher priority than Internet traffic..) that can be solved fairly easy. Or is there something I don't see ???

stolsma
alcatell1
50%
50%
alcatell1,
User Rank: Light Beer
12/5/2012 | 3:44:56 AM
re: AT&T Still Has IPTV 'Jitters'
or possibly fatal flaws in 7450 multicast throughput
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'

Couple of things. You are all making an assumption that any packet loss is due to forwarding issues, prioritization, or congestion.

1 - 1 Packet (potentially 1 Bit Error) in 2 minutes of data transmission is a BER of around 10-9 for an SDTV stream (1 error in approximately 240 Mb). Ever checked the BER of a DSL line?

2 - The buffers are very small on MS STBs to achieve the fast channel change. In normal IPTV setups, there are seconds of buffering and channel switches look like Satellite systems. Network jitter must be minimized to keep these STBs fed. This limits abilities of the multicast network to buffer in replication - as such buffering will add jitter as more paths are added to that channel.

seven
gianconstantine
50%
50%
gianconstantine,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
I think you may not be considering the nature of multicast vs. unicast in delivering video. The bandwidth requirements on the backplane are not heavy enough to cause problems. Consider 250 channels encoded in VC1. While high, let's consider 3Mbps stream rate. Even if all 250 channels are being subscribed to off all cards in 10 blade chassis (for argument's sake), the switching fabric would be pushing less than 8Gbps.

I ran an IPTV network in Atlanta for all of 2005. I slammed Cisco 6500 Sup720/MSFC3 with multicast and unicast traffic and the switch fabric stats were as clean as a whistle.

Microsoft and Alcatel have some sort of funky setup that's screwing them. This problem is not so simple as an incapable switch or router.

Gian.
cabecar
50%
50%
cabecar,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
VoD should not have the same issues as it is not Multicast traffic. The problem with VoD is that each new instance requires a new stream. This means that they need to have Video pump engines close to the edge... if they don't a spike in VoD usage can affect users over the entire network. If they do have pump engines near the edge then a spike will only affect edge users in a particular region. To prevent this they need to implement video CAC for Unicast VoD.
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
Thanks for all the reply's. It's all a little bit clearer to me now..

I know something about the 7450 and I'm almost sure there is nothing wrong with the multicast replication (as sure you can be after 9 months of testing..). And to be honest we didn't see any alarming packet loss (except some bit errors on bad optical lines that resulted in bad packets, resolved by replacing optical cards, can happen any time).

What we did see was packet loss on DSL lines because of impuls noise and bad copper lines (comparable to seven's 10-9), btw some impuls ratio settings and DSL CRC checking can help there. Trade off is the added latency for other traffic like gaming or internet pings. When I wrote my first message I was still under the impression that Lightspeed was FttH but I saw that also FttC (with DSL as last/first mile) was used.

Personally I think that solving this problem with unicast retransmit requests is not the way to go but who am I...

stolsma
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
cabecar wrote: ".... To prevent this they need to implement video CAC for Unicast VoD."

Anyone out there who can clarify if AT&T is doing something like this ??? I know some operators who are trying to implement a function this. Some with great results!!

stolsma
Page 1 / 3   >   >>
From The Founder
Light Reading sits down at CES with the head of Cisco's service provider video business, Conrad Clemson, to discuss how NFV and cloud security relate to video, the challenge of managing 4K/8K traffic, the global expansion of Netflix and virtual reality.
Flash Poll
Live Streaming Video
CLOUD / MANAGED SERVICES: Prepping Ethernet for the Cloud
Moderator: Ray LeMaistre Panelists: Jeremy Bye, Leonard Sheahan
LRTV Documentaries
EANTC Tests Nokia IP Routing & Mobile Gateway VNFs for Real World Deployment

2|9|16   |   5:08   |   (0) comments


Nokia obtained validation of its virtualized router and virtualized mobile gateway capabilities through rigorous testing performed by EANTC. The results set a new industry benchmark for outstanding performance, scalability, resiliency and manageability. Nokia VNFs are ready for telco cloud deployment, so that service providers can accelerate mobile, business and ...
Between the CEOs
CEO Chat With Level 3's Jack Waters

2|8|16   |   26:15   |   (1) comment


Light Reading CEO and founder Steve Saunders sits down with Level 3 Communications' CTO Jack Waters to discuss hot topics like virtualization, 4K and the future of telecom...
LRTV Custom TV
The Composable Telco

2|8|16   |   24:46   |   (0) comments


Heavy Reading's Principal Analyst Caroline Chappell presents the keynote at Light Reading's 2020 Vision Executive Summit in Dublin.
LRTV Custom TV
Join Us at the Digital Operations Transformation Summit

2|4|16   |   03:52   |   (0) comments


The Digital Operations Transformation Summit on February 21, 2016 at the Crowne Plaza Barcelona Fira Centre will bring together 50 senior executives to engage in a unique debate on the opportunities and challenges presented by the transformative evolving digital landscape. RSVP now at events@lightreading.com.
LRTV Custom TV
Making the Test: ADVA Ensemble Connector vs. Open vSwitch

2|4|16   |   01:28   |   (0) comments


Light Reading, in partnership with EANTC, recently tested ADVA's Ensemble Connector, which replaces open vSwitch and offers carrier-grade capability and interoperability. The test results strengthen ADVA's credibility as a provider in the virtualization space.
LRTV Custom TV
Bridging the Gap Between PoCs & Deployment in NFV

2|4|16   |   31:50   |   (0) comments


Charlie Ashton of Wind River presents the keynote at Light Reading's 2020 Vision executive summit in Dublin.
Between the CEOs
CEO Chat With Mike Aquino

2|3|16   |   17:34   |   (0) comments


The former CEO of Overture Networks, Mike Aquino, discusses why truly open virtualization solutions provide service providers with the greatest choice.
Shades of Ray
MWC: Buckle Up for 5G & the IIoT

2|2|16   |   02:28   |   (0) comments


This year's Mobile World Congress looks set to be a 5G land grab and a chance to get down and dirty with the Industrial Internet of Things (IIoT) – but what will the 5G discussions actually be about?
LRTV Custom TV
Case Study: Building China's Next-Gen TV Networks

2|2|16   |   5:01   |   (0) comments


With over 2 billion viewers worldwide, Shenzhen Media Group is one of China's largest content producers. By partnering with Huawei and Sobey, SZMG was able to modernize media operations with the Converged News Center, a production studio that is a model for next-generation workflows.
LRTV Custom TV
Quad Channel Modulator Driver with 46 Gbaud Capability from MACOM

1|28|16   |     |   (0) comments


MACOM's MAOM-003427 is the industry's first surface-mount modulator driver with 46 Gbaud capability to support next generation 200G and 400G applications.
LRTV Custom TV
Video Infographic: Validating Cisco's NFV Infrastructure

1|26|16   |   02:24   |   (1) comment


We all know that the network of the future will be virtual, but when will virtual become a reality? This video infographic covers the four key areas in which Light Reading, in partnership with EANTC, tested Cisco's NFV infrastructure: performance, reliability, multi-service capabilities and single pane of glass management.

For the full report, see

Between the CEOs
CEO Chat With Phil McKinney, CableLabs

1|22|16   |   13:36   |   (1) comment


At CES in Las Vegas, we met with Phil McKinney, CEO of CableLabs. Phil provides an update on the rollout of DOCSIS 3.1, his views on the future of open source and how consumer interest in virtual reality could affect network traffic.
Upcoming Live Events
March 10, 2016, The Cable Center, Denver, CO
April 5, 2016, The Ritz Carlton, Charlotte, NC
May 23, 2016, Austin, TX
May 24-25, 2016, Austin Convention Center, Austin, TX
All Upcoming Live Events
Hot Topics
Google's 5G Radio Ambitions Are Expanding
Dan Jones, Mobile Editor, 2/5/2016
Cincinnati Bell Joins Weight Watchers Club
Alan Breznick, Cable/Video Practice Leader, 2/5/2016
Yahoo & Verizon Sitting in a Tree...
Brian Santo, Senior editor, Test & Measurement / Components, Light Reading, 2/8/2016
Vodafone: Flexible Work Policies Boost Profits
Sarah Thomas, Editorial Operations Director, 2/8/2016
It's Time to Integrate OTT Video
Alan Breznick, Cable/Video Practice Leader, 2/8/2016
Like Us on Facebook
Twitter Feed
Webinar Archive
BETWEEN THE CEOs - Executive Interviews
Light Reading CEO and founder Steve Saunders sits down with Level 3 Communications' CTO Jack Waters to discuss hot topics like virtualization, 4K and the future of telecom...
The former CEO of Overture Networks, Mike Aquino, discusses why truly open virtualization solutions provide service providers with the greatest choice.
Animals with Phones
Retro Is the Way to Be Click Here
Some animals are too cute for smartphones.
Live Digital Audio

Broadband speeds are ramping up across Europe as the continent, at its own pace, follows North America towards a gigabit society. But there are many steps to take on the road to gigabit broadband availability and a number of technology options that can meet the various requirements of Europe’s high-speed fixed broadband network operators. During this radio show we will look at some of the catalysts for broadband network investments and examine the menu of technology options on offer, including vectoring and G.fast for copper plant evolution and the various deployment possibilities for FTTH/B.