Internet Core Router Test

Ever since Juniper introduced its OC192 interface in early 2000, Cisco’s sales force has jumped on a supposed problem with packet reordering. This reordering occurs because there are actually four paths through Juniper’s OC192 card, which means that packets taking different paths can arrive out of sequence.

This subject has become something of a point of honor in the marketing war between the two vendors, with resident experts on both sides explaining why packet reordering is, or is not, an issue.

Since the Smartbits records whether packets arrive in sequence, we were able to determine exactly how much reordering actually does occur and also analyze what impact it may have. (For the record, we should note that packet sequencing was not one of the metrics stated in the original test methodology. But in pursuit of Peace in Our Time, we thought it best to tackle this touchy subject.)

Let’s start by saying that Cisco is right — at least on one count. Juniper’s OC192 interfaces do reorder some packets, both for IP and MPLS traffic (see Figure 11).

When forwarding 40-byte IP packets, Juniper’s OC192 cards reordered at most 0.51 percent of traffic for IP or MPLS. With Imix, packet reordering increased to 2.65 percent over IP and 7.77 percent over MPLS.

We also noticed that reordering occurs on the OC192 cards whenever traffic rates exceed 73 percent of line speed with Imix or 56 percent with IP. There was no reordering whatsoever in the OC48 testbed.

In the world according to Cisco, reordering is a Very Bad Thing for TCP (Transmission Control Protocol) connections, which carry 90 percent or more of all Internet traffic. Cisco notes that TCP expects packets to be received in order. If they’re not, retransmissions can occur, leading to higher latency. If the delays are long enough, connections can time out.

To prove its point, Cisco cites a paper issued by The Institute of Electrical and Electronics Engineers Inc. (IEEE) and the Association for Computing Machinery (ACM). Written by two eminent computer scientists, Jon C.R. Bennett and Craig Partridge, it posits that the probability of a TCP connection on the Internet experiencing packet reordering is greater than 90 percent. The paper goes on to attribute much of this reordering to Digital Equipment Corp. switches in network exchange points. (You can check out the evidence yourself at http://puck.nether.net/cisco-nsp/packet_reordering1.pdf.)

Curiously, Juniper gives customers the same paper to explain away reordering; and it offers four arguments as to why the whole thing is a nonissue.

First, it notes that the reordering we saw was nowhere near the 90 percent-plus reported by Bennett and Partridge.

Second, it says reordering is significant only on a per-connection basis; Internet core circuits carry thousands of concurrent connections. Even if two packets do arrive out of order, Juniper says there’s a very low probability of any two packets belonging to any one connection. The vendor also says that it consciously and very willingly decided to trade off some reordering to gain higher throughput and lower latency for all the connections in the pipe.

Third, Juniper notes that TCP and Spirent Smartbits use different methods to account for packet reordering. It also says Smartbits reports more reordering than any TCP implementation would experience.

Finally, Juniper says the impact of reordering by its OC192 interfaces is not cumulative. In other words, if one OC192 interface puts packets out of order, another one down the line is just as likely to put the packets back in place.

So the war of words continues. Fortunately, Light Reading is prepared to illuminate this controversy:

Reordering can have a very negative impact on TCP connections, dramatically increasing delay.

True.

Reordering can lead to retransmissions, delays, and even connection timeouts. But this begs the question of how much delay is acceptable.

Delay is a function of TCP implementation, link speed, link congestion, device reordering, and many other considerations. The maximum latency numbers presented here can serve as a guideline for how much delay each vendor’s routers will introduce because of reordering and other factors.

Reordering is significant only on a per-connection basis.

True and false.

It’s true that two reordered packets may have an impact on any one connection only if both packets belong to that connection. But it’s equally possible that reordered packets belonging to two different connections may have an impact on both.

Internet core circuits may carry thousands to hundreds of thousands of concurrent TCP connections.

True, as far as it goes.

Juniper’s argument suggests that a pipe handling many TCP connections will interleave packets from each connection (for example, each of 50 connections might be represented by one packet, followed by 49 packets belonging to other connections). In essence, the argument assumes that any risk of reordering will be shared equally by all connections in the pipe.

The trouble is, TCP traffic is inherently bursty, with multiple packets from a given connection typically clumped together. There is no general answer as to how much interleaving will occur on a given TCP link. The key issues here are how much interleaving occurs, and how much distance exists between reordered packets. The answers to those questions will differ on any given network.

TCP connections may experience a lower percentage of reordered packets than we saw in our tests.

True.

If a Smartbits interface receives five packets ordered 1, 2, 4, 3, 5, it records only two packets as being in sequence — packets 1 and 2. In contrast, a TCP receiver would experience only one disruption — packet 3.

This difference in accounting methods does not mean the actual impact on TCP will be three times lower than the Smartbits numbers. Even one disruptive packet can take a long time in arriving, resulting in high latency or, eventually, a connection timeout.

Multiple interfaces will not have a cumulative effective on packet reordering.

False.

If one Juniper OC192 card scrambles some packets, a second OC192 interface has an equal likelihood of correcting the reordering; scrambling the packets further; or making no change. Thus, the impact of multiple OC192s is neither additive nor subtractive.

Perhaps the best way to characterize the reordering issue is to say that its probability is fractal. If one OC192 interface reorders 2 percent of packets, then 100 or 100,000 interfaces have the same probability of reordering 2 percent.

It would be nice to offer a definitive yes or no answer to the whole reordering debate. Nice, but not accurate.

Our test results suggest that Juniper’s OC192 reordering is nowhere near as big a problem as Cisco claims. Nor is it a complete nonissue, as Juniper contends. Since different networks handle different numbers of TCP connections, and since TCP implementations vary widely, we may never be able to completely resolve the question completely.

But there are two statements we can make with certainty (pay attention, they will be on the final):

First, users of Juniper’s OC192 cards won’t experience packet reordering until interface utilization exceeds 73 percent (or 56 percent for those strange few whose traffic consists entirely of 40-byte IP packets).

Second, given the information at hand we can’t definitely prove that reordering will never pose a problem under any circumstance. There’s only one surefire means of eliminating reordering as an issue: Don’t do it in the first place.