Traffic Tracker Helps Trilogy
The free lunch in this instance is Traffic Tracker, which enables Trilogy to monitor traffic on two T1 access lines in real time, free of charge. Using this service, Trilogy has discovered problems that it wasn't previously aware of -- even though it was already monitoring lines with some shareware tools.
Although companies can disguise their identity in the Traffic Tracker project, Trilogy chose not to. It opted to use the service to monitor two T1 access lines, one from AOL Time Warner Inc. (NYSE: AOL) and the other from MegaPath Networks Inc.
After Trilogy had activated Traffic Tracker monitoring, Darren Galatas, in charge of Internet support, didn't pay much attention to it. Galatas already uses a set of shareware tools -- RRDtool and Cacti -- and he probably thought that anything that was free, like Traffic Tracker, wasn't going to add much to what he already knew about his access lines.
After the guided tour by Light Reading's partner in the Traffic Tracker project, Net Evidence, Galatas perked up. "The extra information is very neat" he said. "Seeing the extra metrics like queueing, dropped packets, and line outages is really useful."
Traffic Tracker showed that Trilogy's Time Warner circuit is discarding a lot of packets during busy periods, which may be slowing down user connections. Galatas says he plans to make some configuration changes to his router to try and eliminate the problem.
What Galatas is refering to is reproduced below -- a chart on Traffic Tracker that summarizes packet discards, queue delays, line errors, and availability for a week in the life of one of Trilogy's access lines. Problems are shown as vertical marks, the severity being indicated by the color -- cold (blue) indicating minor problems and hot (red) indicating major ones.
As can be seen, Trilogy's line was up all the time, only suffering one minor line error on Thursday. But traffic on the line is suffering delays, particularly on Thursdays and Fridays. On packet discards, things are so bad that the vertical marks run together forming solid red bars -- something that will be causing a lot of timeouts and probably making user sessions drag and stutter.
The reason for these problems is evident from another Traffic Tracker chart, below, showing traffic volumes on the same access line for the same week:
The vertical axis is the percentage loading on the T1, and each stripe represents a day, so you can see at a glance how days stack up against each other. Traffic levels are color coded so it's a snap to pick out heavy periods -- they're the orange and red areas. On Thursday and Friday the line is maxed out ("soaked" or "wedged" in technical parlance) almost continuously during the working day.
Interestingly, Traffic Tracker's reporting also revealed that traffic wasn't balanced across the two lines, as Galatas had expected: While Trilogy's Time Warner circuit is running flat out, the Megapath line is running pretty quiet:
Adjusting the balance of traffic would definitely relieve a lot of the congestion, improve the users' experiences, and likely delay the expense of an upgrade.
Overall, Trilogy makes heavy use of its Internet connectivity. Traffic Tracker shows that occasionally someone generates a huge volume of traffic that saturates at least one link for several hours at a time. Here's outgoing traffic for the day of January 15th, for example :
The lines carry a wide variety of traffic, so it would normally be hard to pin down the culprit, but Galatas runs other logging systems that he says will let him identify the source of the spikes.
At least we know where this one came from:
It's for Saturday January 26th. The SQL worm knocked on the door for six hours -- loading the line so hard that Traffic Tracker couldn't always collect statistics -- before deciding it wasn't going to get in.
In fact, Traffic Tracker helped Light Reading figure out which service providers were worst hit by the Slammer worm, demonstrating the power of collecting statistics from access lines (see 'The Internet Has Broken').
Overall, Trilogy is really punishing its Internet access. Traffic Tracker shows two lines that are fully loaded for hours at a stretch. Galatas rates Trilogy's users as a demanding and technically aware group, currently happy with their Internet connectivity. This is surprising, given the congestion -- but if degradation happens very slowly over a long period, users' expectations often keep pace with it. And if the applications they're running are very tolerant of delays and lost packets, they'll keep plugging along.
All the same, a line upgrade is going to be necessary before too long, and when that happens, Galatas's users might get a pleasant surprise.
To see what's happening on Trilogy's lines right now, click on this link. You'll see what all the Traffic Tracker participants are up to.
In participating in the Traffic Tracker project, Trilogy is also helping users in general. It's enabling Light Reading to aggregate and analyze traffic statistics from lots of access lines, in a way that promises to provide unique insights into traffic trends on the Internet. Those insights include performance comparisons, which should encourage operators to improve the services they offer enterprise users (see Traffic Tracker Project Is Ready to Roll).
More than 150 companies have expressed an interest in taking part in the scheme, although only a handful of them have so far made the minor change in their access router to enable traffic statistics to be collected. The statistics are collected every three minutes, a process that uses negligible bandwidth and poses no security threat.
Enterprise users interested in participating in the project should click on this link to register.
— Richard Thomas, special to Light Reading