Woody Allen once said that 80 percent of life is showing up. That certainly rings true for last month's carrier-grade IPSec VPN test, paid for by Light Reading and run by Network Test Inc. with Spirent Communications test equipment (see Carrier-Class IPSec: the Bigger the Better).

When we asked ten vendors to step up and prove that their IPSec VPN devices were up to the carrier test, only recently public NetScreen Technologies Inc. (Nasdaq: NSCN) and startup Quarry Technologies Inc. showed up (see Quarry Wins LR VPN Test). Although Quarry edged out NetScreen in the test, both showed impressive results, with throughput rates ten times faster than most CPE-based IPSec gateways. They set new speed records, ran at Gigabit Ethernet line rates, and scaled to support thousands of concurrent tunnels.

Notable industry players such as Cisco Systems Inc. (Nasdaq: CSCO), CoSine Communications Inc. (Nasdaq: COSN), Lucent Technologies Inc. (NYSE: LU), and Nortel Networks Corp. (NYSE/Toronto: NT) were invited to the test but chose not to participate. The reasons were varied, but most vendors declined either because the test methodology was not to their liking or because they didn't want to allocate resources to the project (see No Shows).

"The primary reason that CoSine chose not to participate, was that it didn't want to test for just a single customer instance," a spokesperson for the company says.

“We didn’t have the resources to commit during that time frame,” Ann Fuller, a spokesperson for Nortel says, “because the resources were being used on customer trials.”

Despite the excuses, such actions may not be perceived well by the public. A past Research Poll by Light Reading indicates that 39% of readers surveyed believe that when vendors don't show up for tests, it's usually because they're afraid they won't perform well.

The test concentrated on IPSec-based VPNs (virtual private networks), because the alternatives, Multiprotocol Label Switching (MPLS)-based VPNs and the Internet Engineering Task Force (IETF)'s MPLS Martini extensions, are simply not secure enough. IPSec VPNs, on the other hand, offer strong security. Still, finding ones that are carrier-class can be a challenge. Most IPSec VPNs are intended for use on customer premises equipment (CPE) and don't come close to scaling to carrier-class levels.

Both Quarry's iQ4000 and the NetScreen-5200 proved to be up to the test, offering high levels of security and carrier-class performance. Of course, there were differences between devices -- not least, their intended use. NetScreen’s device is a purpose-built VPN gateway, while Quarry’s box is a switch/router that happens to support IPSec. The test found pros and cons with each device. While the 5200 cost less than the iQ4000 and set up far more concurrent tunnels (see Scaling Up), Quarry finally stole the show with higher throughput in most tests, full redundancy of components, and an intuitive, powerful management platform (see Inside the CO, Speed Demons, Management Material).

The test of the two gateways was grueling. By pounding them with traffic from a Spirent SmartBits analyzer with Gigabit Ethernet interfaces and asking them to send various packet sizes at gigabit speed, the test aimed to get at the devices' real performance numbers. When bragging about performance, many vendors tout throughput numbers obtained in less-than-stressful network situations (see Lies, Damn Lies, and Vendor Specs). It is, for instance, easier to achieve high throughput by using larger packets and weaker encryption and message authentication.

Both devices set new speed records, but one of the major differentiators in the test was the throughput with 1,518-byte frames. While Quarry managed a throughput of 875.3 Mbit/s, NetScreen’s throughput dropped to only 276.6 Mbit/s. In addition, Quarry’s results were by far the most impressive when handling 64-byte frames, which it moved at 540.0 Mbit/s -- close to the theoretical maximum rate for IPSec in ESP tunnel mode.

“This suggests that Quarry gateways can handle short frames -- and transaction-intensive applications that use them, such as databases -- with no throughput penalty,” David Newman, president of Network Test, writes in the report.

All in all, Quarry notched up an impressive victory, but Newman also gave props to NetScreen for making it to the test and showing some good results.

— Eugénie Larson, Reporter, Light Reading
http://www.lightreading.com