Video services

AT&T Still Has IPTV 'Jitters'

Industry sources say AT&T Inc. (NYSE: T) is struggling with video packet loss on the eve of the major market launch of its U-verse IPTV service.

Word has it the U-verse network loses roughly two packets of data per minute. “A lost video packet is more than 1,400 bytes of information, and that's going to cost you a half second of video,” one source says. For the viewing public that can mean little annoyances like screen pixelation and jitter -- or, at worst, full screen freezes.

In the U-verse distribution network, video packets hop from AT&T’s video super headend, to regional headends, to the local central offices, to nodes in the neighborhoods. At each "hop," packets can arrive in incorrect order or overload the buffers within the routers and switches, leading to losses.

An AT&T spokesman chose not to comment on the packet loss issue.

Microsoft Corp. (Nasdaq: MSFT), which supplies the IPTV middleware for the U-verse network, is said to be hard at work increasing its product's capacity for dealing with lost packets. Using a software algorithm called Resilient UDP, the set-top box, upon detecting missing or misplaced packets, sends a "resend" command up through the network. The missing packets are then sent down from a server. (See Microsoft Soups Up the Set-Top.)

That works fine if traffic levels are normal, the source says. But if packet loss should occur during the final minutes of the Super Bowl, a million resend requests could pummel the network at the same time, seriously burdening the system.

One source close to the situation says Microsoft has already built in a 15 to 30 second delay to live video streams to allow some time for dealing with packet loss. AT&T, the source says, is uneasy about the scaleability of the setup.

Microsoft TV Edition product manager Jim Baldwin says his company's middleware platform adds roughly a quarter of a second delay for packet error correction and another second of delay for instant channel changing, but that's it. Baldwin says AT&T is perfectly happy with his company’s Resilient UDP approach to packet loss, but says AT&T may decide to use forward error correction as well.

The Scientific-Atlanta Inc. encoders at the headend of the system, Baldwin says, will add a certain amount of latency to the video streams, but AT&T will decide how much. Scientific-Atlanta chose not to comment for this story. (See AT&T, Verizon Tout Telco TV .)

Meanwhile, AT&T may be hedging its bets. Sources close to the situation say AT&T engineers are experimenting with forward error correction (FEC) on one leg of AT&T's video network. Applied near the headend encoders, FEC adds additional video bitstreams that can be used to reconstruct damaged streams on the fly, downstream in the network. Also at the headend, the video packets are tagged sequentially so that the system can detect interruptions in the packet order downstream. The FEC technology monitors the bitstreams at several points in the network, including the set-top box, to detect missing or damaged packets.

Baldwin says FEC works well when the type and size of the packet loss is predictable. But IP networks can lose a whole packet, groups of consecutive packets, or just one bit within a packet, he says.

U-verse was already under criticism, as analysts have noted the initial Project Lightspeed rollout of broadband access won't have the punch to carry high-definition TV, and some question whether Lightspeed will roll out as quickly as AT&T hopes. (See Is Lightspeed Slowing?)

Flickers of trouble?
U-verse debuted in late June in AT&T’s home turf of San Antonio, Texas, and the AT&T spokesman says the service is getting good reviews from users. The few actual U-verse users contacted by Light Reading say the packet loss issue, at least so far, hasn’t been very visible on their TVs. (See AT&T to Launch Lightspeed Next Month.)

“The flicker [pixelation] has only appeared a few times,” says U-verse user Alan Weinkrantz. “But keep in mind that I also have HD Cable from TimeWarner, and there are times when I also get flicker or a nano-of-a-second blur on that.” Weinkrantz, whose day job is as a technology public relations man, has become a minor celebrity through his U-verse user's blog.

“I had some pixelation the very first day after I got service,” writes another U-verse customer, Chad Brantly. “The next day, I got a call from AT&T. They said that they had been having some problems in my area, but they had just done a hardware update and things should be better. Since then, I haven't seen any pixelation. The service has been great,” Brantly writes.

A few unhappy U-verse adopters, however, have shown up on a bulletin board called Uverseusers.com. One of the five discussion threads on the site is called “pixelations and freeze-ups,” wherein three users -– “eapinsatx,” “dilbert” and “nohbdy” -– complain of moderate to serious pixelation and screen freezes.

Heavy Reading analyst Rick Thompson points out the San Antonio debut is probably happening in a very "controlled" network environment. With the technology world watching, AT&T is surely taking steps to make sure the fledgling service makes a good first impression.

But the issue of scaleability looms for AT&T’s engineers, as the carrier plans to roll out U-verse in 15 to 20 markets by the end of 2006. (See AT&T Readies Lightspeed in North Texas.) "It's one thing to get a complex technology like IPTV rolled out to a group of a few thousand users," says independent telecom analyst Kermit Ross. "But it’s quite another thing to kick that up a notch to a few hundred thousand and then yet another thing to kick it up to a few million users."

In other words, once AT&T cranks up the numbers, any failure to tame packet loss will be evident on millions of TV screens.

— Mark Sullivan, Reporter, Light Reading

lazer-beam 12/5/2012 | 3:42:18 AM
re: AT&T Still Has IPTV 'Jitters' I checked these guys out at NAB..they are the real deal...talking to IBM, Grass Valley, Warner, Bluefish, BBC, and more...maybe ATT needs to call them.....Lazer

> Up to 16x Faster !! Encoding Delivers Live 2-Pass VBR Broadcasting
> San Marcos, TX (August 31, 2006) Kula Media Group, Inc announced
> today that it will roll out its new encoding product lines in October
> 2006. Product demonstrations and pre-orders will be made available at
> this year?s International Broadcaster?s Convention (IBC) in Amsterdam,
> Netherlands, September 8 th -12 th at booth #7.419.
> Products will include the much anticipated, KulaByte TM Broadcaster,
> KulaByte TM Professional and KulaByte TM Personal Encoders.
> First unveiled at the 2005 IBC show in Amsterdam, the KulaByte TM
> technology demonstrated its capability to take full advantage of multi
> core technology to enable increased speed.
> Additionally, the KulaByte TM encoding process increases video quality
> and reduces bandwidth consumption of streaming video over various
> encoding platforms to include H.264,On2?s VP6, Microsoft?s WM9,
> MPEG-4, MPEG-2, and VC1.
> With KulaByte TM Broadcaster, speed tests prove that encoding times
> are up to 16x faster than standard encoding methods for SD and HD
> video. Speeds up to 4x and 2x can be expected with KulaByte TM
> Professional and Personal editions respectively. All KulaByte TM
> products achieve increased qualities and reduced bandwidth consumption
> of streaming video up to 30%.
> KulaByte TM products will feature live 2-Pass and 4-pass VBR encoding.
> KulaByte TM is the only streaming solution that allows for Live
> 2-Pass VBR due to it?s patent pending TimeSlicingTM technology. Live
> broadcasting at, and beyond video on-demand qualities is now a reality
> withKulaByte TM .

> About Kula Media Group : Kula Media Group, Inc. is a leading developer
> of advanced video processing technologies.
> Based in San Marcos, TX, Kula?s engineering and management team is focused on
> delivering the fastest Live and On-Demand video encoding solutions
> while achieving the
> highest video qualities for streaming and archival.
> Visit www.kulabyte.com for more information
> Editorial Contacts
> Stuart English
> [email protected]
> Phone +1 512-535-1183
paolo.franzoi 12/5/2012 | 3:43:57 AM
re: AT&T Still Has IPTV 'Jitters'
Good lord, now that is funny.

I guess the DSL lines that they put out never have errors. Interesting how that Optical network ad has fixed that.

Interesting as well how this DTM has stopped all cable cuts and product outages on all products on the network.

taklax 12/5/2012 | 3:43:57 AM
re: AT&T Still Has IPTV 'Jitters' Unfortunately, packet switched networks will never deliver 100% QoS, so there will always be problems when using them for atreaming services (TV). Packet switching was made for data transmission without requirement on quality (Best Effort).

If you ask Matanuska Telephone Company or Vernon Telephone, who are both using the DTM-technology invented by Swedish Net Insight, (www.netinsight.net) they have never heard of packet loss or jitter in their networks!
lml2000 12/5/2012 | 3:44:06 AM
re: AT&T Still Has IPTV 'Jitters' The scalability issues were evident with MSFT before T started Lightspeed. Presume u'r familiar with the issues of Swiss Telecom's roll out of IPTV, which also runs on MSFT's platform. Same scalability problems cited over a year ago. Don't recall reading any updates on problems exp'd a year ago.

Sorry, but I have to question the wisdom of SBC's/T's senior management in going with MSFT. Heck, they can't even build a reliable OS for a PC. And SBC goes ahead & inks a deal w/MSFT to trust its software to run a network on such a large scale with unprecedented traffic demands?What were they thinking?
Ferdinand 12/5/2012 | 3:44:06 AM
re: AT&T Still Has IPTV 'Jitters' Not only small Telcos in the US do not seem to have problems with IPTV.
Belgacom offers the service also and it is running ok.
What is the problem with AT&T? Or is it Microsoft?
opticalwatcher 12/5/2012 | 3:44:16 AM
re: AT&T Still Has IPTV 'Jitters' Whether occasional pixelization/screen freeze is just an annoyance or is completely unacceptable depends on bad it is and how often it happens.

Surely there's a degradation of quality that would drive even the most unflappable customer crazy.

So just how bad is it?

Also, small telcos such as SureWest have been doing IPTV for a while, including HDTV. So certainly IPTV is do-able. Do they have these problems?
gianconstantine 12/5/2012 | 3:44:32 AM
re: AT&T Still Has IPTV 'Jitters' You would be surprised, but screen freezes are worse than a complete channel outage. Screen freezes offer the hope of watching a channel and tend to piss off the customer even more than a black screen. An already annoyed customer is more likely to complain about screen pixelation, too.

eewhiz 12/5/2012 | 3:44:36 AM
re: AT&T Still Has IPTV 'Jitters' In perspecitve here...
Remember, a major percentage of US subscribers are Direct/Dish subscribers, and they have to put up with occasional pixelizations, but worse yet, with rain-fade, many have to put up with complete outages. If the subscriber base has not complained about that (turn their DISHes back-in), then they won't get too upset by occasional pixelizations or arbritrary screen freezes...
heritagejd 12/5/2012 | 3:44:38 AM
re: AT&T Still Has IPTV 'Jitters' Can't solve everything with software!
heritagejd 12/5/2012 | 3:44:38 AM
re: AT&T Still Has IPTV 'Jitters' Frank- You are very perceptive, it's all in the "Last Mile"!
heritagejd 12/5/2012 | 3:44:38 AM
re: AT&T Still Has IPTV 'Jitters' Honest- Try turning on four(4) hdtv's and see how many streaming channels you get, and quality of defintion. It's all Bandwidth and 20-25 mbs is not ample!
Frank 12/5/2012 | 3:44:49 AM
re: AT&T Still Has IPTV 'Jitters' I'm willing to bet that there are many MSOs whose networks are suffering similar ills as they continue to pump up their HDTV commitments and provisions, and increase their Internet access speeds and their digital content, possibly more so during peak times than at other times.

Of course, AT&T's measly last mile design, and its insistance to stay with copper, has forced certain constraints that wind up being additive across other network elements in the IPTV platform, but the problem is not due to any of those parameters alone.

From my perspective, one thing is for sure -- and I speak from my own experiences, only -- and that is, the problem is definitely increasing, even on HFC systems, instead of getting better. And this makes eminent sense, when you think abou it, considering the trending taking place marked by continually adding more load to most systems.

In Brooklyn, NY I frequently encounter pixelation on my cable service, which is often followed by a momentary dropout and re-synching on identifiable "digital channel groupings". I.e., the six or ten NTSC 6MHz channels that have been compressed into a single digital channel occupying 6MHz of bandwidth in the digital spectrum of the TW system serving me.

The root causes of the problems that AT&T is encountering - which may be a combination of Microsoft's IPTV setup and attempting to provide triple play services on a bandwidth-restricted platform, among other causes - differs (or only partially overlaps) with the problems that some MSOs are, or very soon will be, facing.

And this points to a pet peeve of mine. I.e., there is no means in place that allows one to obtain a snapshot of how well his or her service is performing, much less assuring the accountability for said performance, that cable tv providers deliver, and this applies to both the MSOs and the Telcos. Just as there is no means for most power utilities to determine who does and who doesn't have power during sporadic power outages.

Ferchrissakes, it's 2006, and the power company still can't tell which of its subscribers is receiving its service, which results in the utility having to walk door to door to find out who does. I want to have access to a dashboard on the Web that I can go to and observe just how much congestion exists on my last mile segment, both for purposes of assessing my Internet access and my tv reception; the number of errored seconds and minutes, selectable per time frame of my choosing, that exists on my cable service for any given channel or group of channels; the status of VoD operation, DVR and other user selectable features; the status of my last reported trouble to the help desk; and more.

This is not only feasible to do, but is being done behind the curtains for the utilities' own consumption! In enterprise communications circles such a window into the carriers' monitoring capabilities for the purpose of assessing SLA performance is known as CNM, or customer network management, which is usually a read-only type of accommodation to let the customer know what is going on.

Getting back to my local cable service, some family members who live about two miles from me, also in Brooklyn, receive Cablevision Communications digital tv service, and they tell me that similar glitches and artifacts show up on their digital channels, as well.

The most entertaining experience I'm having these days occurs when I find it necessary to reboot my STB (take your pick... the one in the living room, the den, or the upstairs study ... but it never happens to all three at once) that seem to lose some of their program elements for VoD features but not for regularly scheduled programming.

Worse, the VoD feature sometimes works for one selection (HBO's The Wire, for example), but not for another (an HBO Boxing event) unless, of course, I remove power to the STB and endure the reboot cycle every time.

Yes, of course I've reported this to TW. But let me ask you this: Do you think that the problems I've described above manifest themselves when the cable guy shows up? Not on your life. And so it goes.
Honestly 12/5/2012 | 3:44:49 AM
re: AT&T Still Has IPTV 'Jitters' Delay in traditional broadcast networks would rock you back a bit. Also, as a Direct TV customer, I get pixelation disrupts randomly.

I hear from users that the ATT service, especially the features are really cool. Instant channel change and a smart guide would make me much happier with my Direct DVR. ATT has this, I do not.

dilberts 12/5/2012 | 3:44:50 AM
re: AT&T Still Has IPTV 'Jitters' The pictures tear really bad and are very annoying to watch. Other times the screen just totally locks up and it takes 15 minutes to reset. That is a real problem in the middle of a game. Nowhere near the viewing experience of satellite or cable. I don't know how they expect to do HDTV if standard definition is unwatchable. Most of the customers are at&t employees so you really can not believe what they say about how good things are going.

The other big problem is the DVR stops recording if you try and change channels? And no surround sound. What the att reps say is that it will all be fixed by the end of the year, but no one can tell you how?

Another interesting point is the DSL service is not any faster than it was before U-verse. Cable is still quite a bit faster and more reliable.
Mark Sullivan 12/5/2012 | 3:44:51 AM
re: AT&T Still Has IPTV 'Jitters' Aw, you're no fun.

But seriously, read the first half of the article. The UverseUsers comments were thrown in just for laughs.
cc_junk 12/5/2012 | 3:44:52 AM
re: AT&T Still Has IPTV 'Jitters' Is this a Lightreading invented problem? Sheesh, they refer to 3 users complaining about service. Perhaps enough for Lightreading to create a "panic" article, but nothing I would take note of.
cbrantly 12/5/2012 | 3:44:52 AM
re: AT&T Still Has IPTV 'Jitters' The author of this story mentions the problems experienced by one user at UverseUsers.com named dilbert. It is worth mentioning that all of dilberts posts have been made from IP addresses owned by TimeWarner's RoadRunner cable internet service. It is very questionable whether or not dilbert actually has U-verse or if he is just a troll.
stolsma 12/5/2012 | 3:44:52 AM
re: AT&T Still Has IPTV 'Jitters' Indeed one of the biggest problem areas: In home wiring. There is also one other area at home that is important: the wire used from the copper hand-off interfacepoint to the DSL modem. Some customers use simply a cheap non twisted copperpair. And then when a big truck/bus/thunderstorm comes around or a local radio transmitter freak starts transmitting, the TV service doesn't work any more... Very strange.. ;-)

btw there was a firmware update (one of lots of others solving other problems, but what do you expect of equipment/software just 2 years old) for better multicast performance in the 7xxx but you had to have a lot of multicast traffic to experience the problems. In real life we didn't see any problems with multicast.

metroman 12/5/2012 | 3:44:53 AM
re: AT&T Still Has IPTV 'Jitters' I have heared a lot of rumour about the 7450 and it's multicast performance. Nothing concrete just the same comments coming up about it "having issues". Can anyone shed any light?

And in reply to the other comment about multicast on switch fabrics following my posting:

3 key points -

1. Multicast is not the only traffic I am sure.

2. It's not about fabric capacity, it's about output queue capacity. When doing multicast replication all output queues have to be available before any of them can transmit. If you have one congested output queue you have an issue and possible packet loss.

3. Switches do not have infinate replication capacity, they all have a limit. Also the replication may not be done very well, or at line rate. These architectural issues can combine to have a detrimental affect.

It may not be the case here, just for clarity.


cbrantly 12/5/2012 | 3:44:53 AM
re: AT&T Still Has IPTV 'Jitters' I am a U-verse customer and have talked with logs of users on the UverseUsers.com message board. Most users have found that their packet loss / jitters are a result of using the existing in-home coax. I had the installer run Cat-5 to my gateway and both of my TVs and I have had 0 packet loss.
stolsma 12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters' Thanks for all the reply's. It's all a little bit clearer to me now..

I know something about the 7450 and I'm almost sure there is nothing wrong with the multicast replication (as sure you can be after 9 months of testing..). And to be honest we didn't see any alarming packet loss (except some bit errors on bad optical lines that resulted in bad packets, resolved by replacing optical cards, can happen any time).

What we did see was packet loss on DSL lines because of impuls noise and bad copper lines (comparable to seven's 10-9), btw some impuls ratio settings and DSL CRC checking can help there. Trade off is the added latency for other traffic like gaming or internet pings. When I wrote my first message I was still under the impression that Lightspeed was FttH but I saw that also FttC (with DSL as last/first mile) was used.

Personally I think that solving this problem with unicast retransmit requests is not the way to go but who am I...

stolsma 12/5/2012 | 3:44:54 AM
re: AT&T Still Has IPTV 'Jitters' cabecar wrote: ".... To prevent this they need to implement video CAC for Unicast VoD."

Anyone out there who can clarify if AT&T is doing something like this ??? I know some operators who are trying to implement a function this. Some with great results!!

paolo.franzoi 12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters'
Couple of things. You are all making an assumption that any packet loss is due to forwarding issues, prioritization, or congestion.

1 - 1 Packet (potentially 1 Bit Error) in 2 minutes of data transmission is a BER of around 10-9 for an SDTV stream (1 error in approximately 240 Mb). Ever checked the BER of a DSL line?

2 - The buffers are very small on MS STBs to achieve the fast channel change. In normal IPTV setups, there are seconds of buffering and channel switches look like Satellite systems. Network jitter must be minimized to keep these STBs fed. This limits abilities of the multicast network to buffer in replication - as such buffering will add jitter as more paths are added to that channel.

gianconstantine 12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters' I think you may not be considering the nature of multicast vs. unicast in delivering video. The bandwidth requirements on the backplane are not heavy enough to cause problems. Consider 250 channels encoded in VC1. While high, let's consider 3Mbps stream rate. Even if all 250 channels are being subscribed to off all cards in 10 blade chassis (for argument's sake), the switching fabric would be pushing less than 8Gbps.

I ran an IPTV network in Atlanta for all of 2005. I slammed Cisco 6500 Sup720/MSFC3 with multicast and unicast traffic and the switch fabric stats were as clean as a whistle.

Microsoft and Alcatel have some sort of funky setup that's screwing them. This problem is not so simple as an incapable switch or router.

cabecar 12/5/2012 | 3:44:55 AM
re: AT&T Still Has IPTV 'Jitters' VoD should not have the same issues as it is not Multicast traffic. The problem with VoD is that each new instance requires a new stream. This means that they need to have Video pump engines close to the edge... if they don't a spike in VoD usage can affect users over the entire network. If they do have pump engines near the edge then a spike will only affect edge users in a particular region. To prevent this they need to implement video CAC for Unicast VoD.
alcatell1 12/5/2012 | 3:44:56 AM
re: AT&T Still Has IPTV 'Jitters' or possibly fatal flaws in 7450 multicast throughput
metroman 12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters' There is another possibility.

If the traffic is being delivered using multicast, the network elements will need to replicate the traffic. Depending upon the hardware in use, it may be that they are having issues either with the replication itself or the forwarding mechanisms for multicast. Ingress replication for multicast may create congestion in switch fabrics, while if the stream is being replicated to many egress interfaces, all interfaces need to be available before the frame is forwarded. In this case it only take one congested interface to cause network-wide packet loss and service degradation.

The network elements need to have robust hardware to deal with multicast replication and well thought out prioritisation & buffering mechanisms.

Does anyone know if they have the same issues on VoD (unicast)?

paolo.franzoi 12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters'
My guess is that the small buffers on the STB to have the short channel switch time is the issue. Pretty much any network imperfections flow through to the STB. Scaling this will be pretty funny, especially that there is no HDTV.

stolsma 12/5/2012 | 3:44:57 AM
re: AT&T Still Has IPTV 'Jitters' I realy don't understand that there are that much packets lost per minute. Packet loss is more or less dependable on congestion and buffer depths, and if the article is true then there is a lot of congestion on the path from Master head end to the Customer. In my opinion a case of bad network planning/packet prioritization (I expect that this video traffic has a higher priority than Internet traffic..) that can be solved fairly easy. Or is there something I don't see ???

trzwuip 12/5/2012 | 3:44:58 AM
re: AT&T Still Has IPTV 'Jitters' does this mean the d-servers dont work the way microsoft promised?
Sign In