Light Reading
Packet loss is plaguing Project Lightspeed's IPTV service, say sources – and a few unhappy IPTV subscribers

AT&T Still Has IPTV 'Jitters'

Light Reading
News Analysis
Light Reading
8/11/2006
50%
50%

Industry sources say AT&T Inc. (NYSE: T) is struggling with video packet loss on the eve of the major market launch of its U-verse IPTV service.

Word has it the U-verse network loses roughly two packets of data per minute. “A lost video packet is more than 1,400 bytes of information, and that's going to cost you a half second of video,” one source says. For the viewing public that can mean little annoyances like screen pixelation and jitter -- or, at worst, full screen freezes.

In the U-verse distribution network, video packets hop from AT&T’s video super headend, to regional headends, to the local central offices, to nodes in the neighborhoods. At each "hop," packets can arrive in incorrect order or overload the buffers within the routers and switches, leading to losses.

An AT&T spokesman chose not to comment on the packet loss issue.

Microsoft Corp. (Nasdaq: MSFT), which supplies the IPTV middleware for the U-verse network, is said to be hard at work increasing its product's capacity for dealing with lost packets. Using a software algorithm called Resilient UDP, the set-top box, upon detecting missing or misplaced packets, sends a "resend" command up through the network. The missing packets are then sent down from a server. (See Microsoft Soups Up the Set-Top.)

That works fine if traffic levels are normal, the source says. But if packet loss should occur during the final minutes of the Super Bowl, a million resend requests could pummel the network at the same time, seriously burdening the system.

One source close to the situation says Microsoft has already built in a 15 to 30 second delay to live video streams to allow some time for dealing with packet loss. AT&T, the source says, is uneasy about the scaleability of the setup.

Microsoft TV Edition product manager Jim Baldwin says his company's middleware platform adds roughly a quarter of a second delay for packet error correction and another second of delay for instant channel changing, but that's it. Baldwin says AT&T is perfectly happy with his company’s Resilient UDP approach to packet loss, but says AT&T may decide to use forward error correction as well.

The Scientific-Atlanta Inc. encoders at the headend of the system, Baldwin says, will add a certain amount of latency to the video streams, but AT&T will decide how much. Scientific-Atlanta chose not to comment for this story. (See AT&T, Verizon Tout Telco TV .)

Meanwhile, AT&T may be hedging its bets. Sources close to the situation say AT&T engineers are experimenting with forward error correction (FEC) on one leg of AT&T's video network. Applied near the headend encoders, FEC adds additional video bitstreams that can be used to reconstruct damaged streams on the fly, downstream in the network. Also at the headend, the video packets are tagged sequentially so that the system can detect interruptions in the packet order downstream. The FEC technology monitors the bitstreams at several points in the network, including the set-top box, to detect missing or damaged packets.

Baldwin says FEC works well when the type and size of the packet loss is predictable. But IP networks can lose a whole packet, groups of consecutive packets, or just one bit within a packet, he says.

U-verse was already under criticism, as analysts have noted the initial Project Lightspeed rollout of broadband access won't have the punch to carry high-definition TV, and some question whether Lightspeed will roll out as quickly as AT&T hopes. (See Is Lightspeed Slowing?)

Flickers of trouble?
U-verse debuted in late June in AT&T’s home turf of San Antonio, Texas, and the AT&T spokesman says the service is getting good reviews from users. The few actual U-verse users contacted by Light Reading say the packet loss issue, at least so far, hasn’t been very visible on their TVs. (See AT&T to Launch Lightspeed Next Month.)

“The flicker [pixelation] has only appeared a few times,” says U-verse user Alan Weinkrantz. “But keep in mind that I also have HD Cable from TimeWarner, and there are times when I also get flicker or a nano-of-a-second blur on that.” Weinkrantz, whose day job is as a technology public relations man, has become a minor celebrity through his U-verse user's blog.

“I had some pixelation the very first day after I got service,” writes another U-verse customer, Chad Brantly. “The next day, I got a call from AT&T. They said that they had been having some problems in my area, but they had just done a hardware update and things should be better. Since then, I haven't seen any pixelation. The service has been great,” Brantly writes.

A few unhappy U-verse adopters, however, have shown up on a bulletin board called Uverseusers.com. One of the five discussion threads on the site is called “pixelations and freeze-ups,” wherein three users -– “eapinsatx,” “dilbert” and “nohbdy” -– complain of moderate to serious pixelation and screen freezes.

Heavy Reading analyst Rick Thompson points out the San Antonio debut is probably happening in a very "controlled" network environment. With the technology world watching, AT&T is surely taking steps to make sure the fledgling service makes a good first impression.

But the issue of scaleability looms for AT&T’s engineers, as the carrier plans to roll out U-verse in 15 to 20 markets by the end of 2006. (See AT&T Readies Lightspeed in North Texas.) "It's one thing to get a complex technology like IPTV rolled out to a group of a few thousand users," says independent telecom analyst Kermit Ross. "But it’s quite another thing to kick that up a notch to a few hundred thousand and then yet another thing to kick it up to a few million users."

In other words, once AT&T cranks up the numbers, any failure to tame packet loss will be evident on millions of TV screens.

— Mark Sullivan, Reporter, Light Reading

(30)  | 
Comment  | 
Print  | 
Newest First  |  Oldest First  |  Threaded View
Page 1 / 3   >   >>
trzwuip
50%
50%
trzwuip,
User Rank: Light Beer
12/5/2012 | 3:44:58 AM
re: AT&T Still Has IPTV 'Jitters'
does this mean the d-servers dont work the way microsoft promised?
metroman
50%
50%
metroman,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
There is another possibility.

If the traffic is being delivered using multicast, the network elements will need to replicate the traffic. Depending upon the hardware in use, it may be that they are having issues either with the replication itself or the forwarding mechanisms for multicast. Ingress replication for multicast may create congestion in switch fabrics, while if the stream is being replicated to many egress interfaces, all interfaces need to be available before the frame is forwarded. In this case it only take one congested interface to cause network-wide packet loss and service degradation.

The network elements need to have robust hardware to deal with multicast replication and well thought out prioritisation & buffering mechanisms.

Does anyone know if they have the same issues on VoD (unicast)?

Metroman
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'

My guess is that the small buffers on the STB to have the short channel switch time is the issue. Pretty much any network imperfections flow through to the STB. Scaling this will be pretty funny, especially that there is no HDTV.

seven
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
I realy don't understand that there are that much packets lost per minute. Packet loss is more or less dependable on congestion and buffer depths, and if the article is true then there is a lot of congestion on the path from Master head end to the Customer. In my opinion a case of bad network planning/packet prioritization (I expect that this video traffic has a higher priority than Internet traffic..) that can be solved fairly easy. Or is there something I don't see ???

stolsma
alcatell1
50%
50%
alcatell1,
User Rank: Light Beer
12/5/2012 | 3:44:56 AM
re: AT&T Still Has IPTV 'Jitters'
or possibly fatal flaws in 7450 multicast throughput
paolo.franzoi
50%
50%
paolo.franzoi,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'

Couple of things. You are all making an assumption that any packet loss is due to forwarding issues, prioritization, or congestion.

1 - 1 Packet (potentially 1 Bit Error) in 2 minutes of data transmission is a BER of around 10-9 for an SDTV stream (1 error in approximately 240 Mb). Ever checked the BER of a DSL line?

2 - The buffers are very small on MS STBs to achieve the fast channel change. In normal IPTV setups, there are seconds of buffering and channel switches look like Satellite systems. Network jitter must be minimized to keep these STBs fed. This limits abilities of the multicast network to buffer in replication - as such buffering will add jitter as more paths are added to that channel.

seven
gianconstantine
50%
50%
gianconstantine,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
I think you may not be considering the nature of multicast vs. unicast in delivering video. The bandwidth requirements on the backplane are not heavy enough to cause problems. Consider 250 channels encoded in VC1. While high, let's consider 3Mbps stream rate. Even if all 250 channels are being subscribed to off all cards in 10 blade chassis (for argument's sake), the switching fabric would be pushing less than 8Gbps.

I ran an IPTV network in Atlanta for all of 2005. I slammed Cisco 6500 Sup720/MSFC3 with multicast and unicast traffic and the switch fabric stats were as clean as a whistle.

Microsoft and Alcatel have some sort of funky setup that's screwing them. This problem is not so simple as an incapable switch or router.

Gian.
cabecar
50%
50%
cabecar,
User Rank: Light Beer
12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
VoD should not have the same issues as it is not Multicast traffic. The problem with VoD is that each new instance requires a new stream. This means that they need to have Video pump engines close to the edge... if they don't a spike in VoD usage can affect users over the entire network. If they do have pump engines near the edge then a spike will only affect edge users in a particular region. To prevent this they need to implement video CAC for Unicast VoD.
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
Thanks for all the reply's. It's all a little bit clearer to me now..

I know something about the 7450 and I'm almost sure there is nothing wrong with the multicast replication (as sure you can be after 9 months of testing..). And to be honest we didn't see any alarming packet loss (except some bit errors on bad optical lines that resulted in bad packets, resolved by replacing optical cards, can happen any time).

What we did see was packet loss on DSL lines because of impuls noise and bad copper lines (comparable to seven's 10-9), btw some impuls ratio settings and DSL CRC checking can help there. Trade off is the added latency for other traffic like gaming or internet pings. When I wrote my first message I was still under the impression that Lightspeed was FttH but I saw that also FttC (with DSL as last/first mile) was used.

Personally I think that solving this problem with unicast retransmit requests is not the way to go but who am I...

stolsma
stolsma
50%
50%
stolsma,
User Rank: Light Beer
12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters'
cabecar wrote: ".... To prevent this they need to implement video CAC for Unicast VoD."

Anyone out there who can clarify if AT&T is doing something like this ??? I know some operators who are trying to implement a function this. Some with great results!!

stolsma
Page 1 / 3   >   >>
Educational Resources
sponsor supplied content
Educational Resources Archive
Flash Poll
From The Founder
It's clear to me that the communications industry is divided into two types of people, and only one is living in the real world.
LRTV Custom TV
Distributed Access Architectures – 2

10|21|14   |   8:51:00 AM   |   (0) comments


ARRIS CTO Network Solutions Tom Cloonan discusses why many if not most MSOs will continue with integrated CCAP, while addressing why some are also looking at two futuristic, distributed access architectures: Remote PHY and Remote CCAP.
LRTV Custom TV
Distributed Access Architectures – 1

10|21|14   |   9:01   |   (0) comments


SCTE Sr. Director of Engineering Dean Stoneback discusses the pros and cons of distributed access architecture (DAA) and its various forms, which range from basic Remote PHY to full CMTS functionality in the node.
LRTV Custom TV
The WiFi Road to Riches – 2

10|21|14   |   3:58   |   (0) comments


ARRIS Senior Solution Architect Eli Baruch talks about how MSOs can enable public and community WiFi through 1) outdoor access points, 2) businesses seeking to offer WiFi to customers, and 3) residential WiFi gateway extensions.
LRTV Custom TV
The WiFi Road to Riches – 1

10|21|14   |   10:15   |   (0) comments


SCTE Director of Advanced Technologies Steve Harris discusses WiFi deployments, drivers, challenges and advances, including 802.11ac, carrier-grade WiFi, community WiFi, Hotspot 2.0, Passpoint, WiFi-First and voice-over-WiFi.
LRTV Custom TV
Advantech Accelerates 100G Traffic Handling

10|17|14   |   7:56   |   (0) comments


Paul Stevens from Advantech explains why handling 100GbE needs a whole new platform design approach and how Advantech is addressing the needs of equipment providers and carriers to give them the flexibility and performance they will need for SDN and NFV deployment.
LRTV Huawei Video Resource Center
Holland's Imtech Traffic & Infra Discusses Huawei's ICT Solution and Services

10|16|14   |   4:49   |   (0) comments


Dimitry Theebe is from the business unit at Imtech Traffic & Infra which delivers communications solutions for transportations. His partnershp with Huawei began about a years ago. In this video, Theebe speaks more about this partnership and what he hopes to accomplish with Huawei.
LRTV Huawei Video Resource Center
Huawei's Comprehensive Storage Solutions Vital for SVR

10|16|14   |   6:16   |   (0) comments


SVR Information Technology provides cloud services for academic and special sectors. With Huawei's support, SVR and Yildiz Technical University has established Turkey's largest and most advanced High Performance Computing system. CSO Ismail Cem Aslan talks about what he hopes Huawei's OceanStor storage system will bring for him.
LRTV Huawei Video Resource Center
Mexico's Servitron's Impression of Huawei at CCW 2014

10|16|14   |   6:35   |   (0) comments


Servitron is a network operator in Mexico that has been in the trunking industry for the past 20 years. Its COO, Ing. Ragnar Trillo O., explains at Critical Communications World 2014 that his company has been interested in the long-term evolution of LTE technology and its adoption for TETRA.
LRTV Huawei Video Resource Center
Building a Better Dubai

10|16|14   |   2:06   |   (0) comments


Abdulla Ahmed Al Falasi is the director of commercial affairs, a telecommunications coordinator for the government of Dubai. Their areas of service span across multiple industries, including police, safety, shopping malls and more. In this video, Abdulla talks about his department's work with Huawei.
LRTV Huawei Video Resource Center
Huawei Lights Up Malaysia Partner Maju Nusa

10|16|14   |   1:59   |   (0) comments


Malaysia's Maju Nusa is an enterprise partner to Huawei in networking, route switches and telco equipment. At this year's Critical Communications World in Singapore, CTO Pushpender Singh talks about what Huawei's eLTE solutions mean to his company and for Malaysia.
LRTV Custom TV
Evolving From HFC to FTTH Networks

10|15|14   |   2:19   |   (0) comments


Cisco's Todd McCrum delves into the future of cable's HFC plant, examining how DOCSIS 3.1 and advanced video compression will extend its life and how the IP video transition will usher in GPON and EPON over FTTH.
LRTV Custom TV
Exploring the Future of Cable Access

10|15|14   |   6:23   |   (0) comments


Cisco's Brett Wingo looks at where cable access architectures are heading, discussing the impact of DOCSIS 3.1, CCAP, Remote PHY, SDN, virtualization of cable networks and related technologies.
Upcoming Live Events
October 29, 2014, New York City
November 6, 2014, Santa Clara
November 11, 2014, Atlanta, GA
December 2, 2014, New York City
December 3, 2014, New York City
December 9-10, 2014, Reykjavik, Iceland
June 9-10, 2015, Chicago, IL
Infographics
WhoIsHostingThis.com presents six of the world's most extreme WiFi hotspots, enabling the most epic selfies you can imagine.
Hot Topics
Forget the Internet, Brace for Skynet
Stephen Saunders, 10/15/2014
HBO Will Go OTT in 2015
Mari Silbey, Independent Technology Editor, 10/15/2014
Google: Carriers & Cloud Providers Need to Cooperate
Mitch Wagner, West Coast Bureau Chief, Light Reading, 10/16/2014
Could Data Be the New 'Currency'?
Ray Le Maistre, Editor-in-chief, 10/16/2014
iPad Air 2 Lets Users Switch Carriers Any Time
Mitch Wagner, West Coast Bureau Chief, Light Reading, 10/17/2014
Like Us on Facebook
Twitter Feed