Light Reading – with testing partners Spirent and Network Test – conducts its biggest test ever: SSL-based VPN gateways * Who's making them? * What are the performance parameters? * There are no losers!

December 9, 2003

34 Min Read
SSL VPNs: Access Anywhere, Anytime

The best technology, according to an old networking truism, is the one that requires the least amount of change to end-stations. By that measure, SSL-based VPNs are right up there.

In the past year, numerous vendors have come to market with gateways that allow secure access to corporate networks from virtually any Web browser, on any device, anywhere.

Unlike existing virtual private network (VPN) technologies, no changes or complex configurations are needed on client machines. And SSL VPNs come with built-in interoperability: The security mechanism needed for strong authentication and encryption, Secure Sockets Layer (SSL), is already embedded in most Web browsers.

SSL has been around for years, but it’s still very early days for its use as a VPN technology. There hasn’t yet been any public comparison of key metrics like performance, scaleability, and, of course, security.

Until now.

Light Reading has teamed up with its testing partners – Spirent Communications and Network Test – to conduct the industry’s most comprehensive comparison of SSL-based VPN gateways.

For nearly three months we’ve used Spirent’s Avalanche and Reflector traffic generator/analyzers to answer a key question: Can SSL VPN gateways scale up to meet the demands of enterprise networks? To find out, we tested them with the ultimate enterprise application, Microsoft Outlook, as well as running a battery of other performance and scaleability tests.

Eight vendors took part in this evaluation:

That’s a record high turnout for any Light Reading test.

It’s not difficult to understand this test’s popularity: This is a new market not yet dominated by one of the giant firms. As we wrapped up testing, Cisco Systems Inc. (Nasdaq: CSCO) announced SSL support for its VPN gateways. But neither Cisco nor any other vendor is anywhere close to market dominance. This is still very much a market up for grabs.

Our findings are promising, though there’s still lots of work to be done:

  • On scaleability, SSL VPNs get generally good marks. A single gateway can support anywhere from hundreds to tens of thousands of concurrent users. The record holder, NetScaler’s NS 9500, supports up to 58,000 users at one time.

  • All systems support Outlook Web Access (OWA), the Webified version of Microsoft’s Outlook mail client, but there’s lots of variation in how well they do so. The good news: Many products handle huge numbers of OWA users at one time. The bad news: Response times can crawl into the minutes when systems support huge numbers of users. That sort of inconvenience could cause users to go elsewhere, circumventing gateway security.

  • Products that lack hardware-based SSL acceleration are generally underwhelming performers. In our forwarding rate tests, some gateways with hardware-based acceleration pushed data at nearly 500 Mbit/s, while the low-water mark for products doing encryption/decryption in software was less than 1 Mbit/s. One commendable exception is the EX-1500 from Aventail, which ran far faster than other products lacking hardware-assisted encryption/decryption.

  • Vendors do a generally good job of locking down their devices. Our security assessment turned up only a few minor annoyances, chiefly around weak default configurations.

We’re not going to pick an overall test winner or winners in this evaluation, for several reasons. First, different products exhibited different strengths in different tests, and we think it makes a lot more sense to let network managers decide which areas matter most to them, and pick a product accordingly. Second, we want to recognize all the vendors that participated in this test; they all deserve considerable credit for participating. Third, we found something interesting and commendable in every single product we tested.

For example, the Array Networks gateway turned in the lowest response times among appliance-based systems in our Outlook tests. Aventail offers the broadest range of configuration options, while the NetScreen gateway’s user interface is a model of clarity the others should follow. As noted, NetScaler blew the doors off in several of our scaleability tests, while Nortel came out tops among server-based systems in tests of Outlook and concurrent users. PortWise and Symantec were value leaders. Last but not least, Whale Communications combines a novel architecture with strong application-layer access controls.

For those of you keeping score, Table 1 offers a brief outline of which participants performed best in various fields:

Table 1: Best by Test

If this matters most to you�

�check out this
server-based product:

�check out this
appliance-based product:

Features

Aventail

NetScreen

Price

PortWise

NetScreen

Session rate

PortWise

NetScaler

Concurrent users

Nortel

NetScaler

OWA concurrent users

PortWise

NetScaler

OWA transaction rate

Nortel

NetScaler

OWA URL response time

Nortel

Array

OWA page response time

Nortel

Array

OWA under attack

Nortel

NetScaler

Forwarding rate

PortWise

NetScaler

Security assessment

� draw �

� draw �





As another sign that this is still a market in flux, two participants got bought during testing. NetScreen bought Neoteris for nearly $300 million, mostly in stock, while Symantec snared SafeWeb for $26 million in cash. Symantec immediately discontinued the SEA Tsunami product we tested, and says it’s got a more powerful product slated for release in January 2004. We’ll refer to both vendors by their new parent companies in this article. Aventail, for its part, has released a new version, 7.0, that the company says has more features and far higher performance than the version we tested.

Here's a hyperlinked summary of the report:

  • SSL VPN Overview
    Configured in hardware or software, SSL gateways will complement – not replace – IPSec

  • Modus Operandi
    Different access modes – including reverse proxy and agent mode – offer different advantages

  • Performance Tests
    We measure performance 5 ways: session rate; concurrent user capacity, Outlook Web Access handling, with and without DOS attacks; and forwarding rate

  • Maximum Concurrent Users
    We look at how many users can do actual work through an SSL VPN gateway at any given instant

  • Outlook Web Access
    OWA tops the list of applications that benefit from anywhere/anytime access secured by SSL

  • The Waiting Game
    Response-time measurements turn up more differences among products than any other test

  • OWA and DOS
    We launch Smurf, SYN floods, UDP floods, and Xmas tree attacks

  • Forwarding Rate
    We measure the most data each system can pump out while handling the maximum number of users

  • Security Assessment
    Nessus test returns positive overall results

— David Newman is president of Network Test, an independent testing and network design consulting firm in Westlake Village, Calif. His email address is [email protected].

Much of the marketing spin and trade press coverage has positioned the SSL VPN as a replacement for the most popular current VPN technology, IP Security (IPSec). We think that’s misguided. Each technology’s strengths actually play nicely to the other’s weaknesses.

Because SSL VPNs enable access from virtually any Web browser, they’re a natural fit for remote access and extranet applications. Those have been sore spots for IPSec, which (at least for IPv4, the current version of TCP/IP) requires the installation and configuration of special software on all clients. Further, IPSec has been beset by interoperability issues, in most cases forcing single-vendor deployments.

These problems don’t make SSL VPNs a better choice in all cases. SSL VPNs are easiest to deploy for applications viewed through a Web browser. SSL VPNs can support other applications by tunneling them over SSL connections, but, as with any tunneling technology, there may be tradeoffs when it comes to performance and access control.

SSL gateways come in a variety of form factors (see Table 2). The products we tested from Aventail, Nortel, Symantec, and Whale all came supplied on rack-mountable server hardware. The PortWise product is software only, although the company will optionally supply server hardware (including SSL accelerator cards, although the HP Proliant server PortWise supplied for testing did not have one).

Table 2: Selected Vendors of SSL-Based VPN Gateways

Vendor

Product/software version tested

Form factor

100-user license (US list price)

Array Networks Inc.
Campbell, Calif.
408-378-6800
www.arraynetworks.net/globalaccess.htm

Array Secure Proxy (SP)/SP 6.1.1.4

Appliance

$25,000

Aventail Corp.
Seattle
206-215-1111
www.aventail.com/products_services

Aventail EX-1500 SSL VPN appliance/EX-1500 6.4.2

Server

$29,995

NetScaler Inc.
Santa Clara, Calif.
408-987-8700
www.netscaler.com/products/9000series.html

NetScaler 9500/5.0

Appliance

$29,999

NetScreen Technologies Inc.1
Sunnyvale, Calif.
408-962-8200
www.neoteris.com/products/access.html

Neoteris Access Series/3.2.1

Appliance

$14,995

Nortel Networks Corp.
Brampton, Ontario
905-863-0000
www.nortelnetworks.com/products/01/alteon

Nortel Networks Alteon SSL Accelerator 410/4.1

Server

$34,990

PortWise AB
Stockholm
+46-864-25-380
www.portwise.com/products-mvpn.shtml

PortWise mVPN/3.5.2

Software

$8,200

Symantec Corp.2
Cupertino, Calif.
408-517-8000
www.symantec.com

SafeWeb SEA Tsunami/4

Server

$16,500

Whale Communications Ltd.
Fort Lee, N.J.
201-947-9177
www.whalecommunications.com/remote

e-Gap Remote Access Appliance/2.5

Server

$30,000

1. Participated as Neoteris, acquired by NetScreen during testing
2. Participated as SafeWeb, acquired by Symantec during testing



It’s technically accurate to describe Whale’s e-Gap as a server running Windows 2000 Server, but it’s also more than that. The system includes a unique SCSI switch that relays traffic from the external to the internal interfaces without having the two sides physically attached at any time. This is where the “gap” in the product name originates. Like PortWise, Whale supplied a gateway for testing without hardware-based SSL acceleration; the company says it generally will “right-size” a gateway and supply acceleration as needed.

The Array, NetScreen, and NetScaler devices are appliances. Appliance vendors say they’ve optimized performance with hardware acceleration and other techniques. Also, these devices have fixed hardware configurations.

Server- and appliance-based systems also differ in price. As tested, all three of the appliances we evaluated have U.S. list prices of $50,000. That’s higher than all the server-based systems we tested. At the low end of the spectrum is PortWise’s mVPN, which comes in at $15,240, including $3,000 for a 1U rack-mount server.

We should note that our “price as tested” will be on the high end of the scale, since we asked vendors to supply products licensed for the largest number of users possible. In many cases, we exceeded those limits on our test bed. We’re presenting 100-user license pricing here.

All products we tested except the Symantec gateway can send traffic over copper Gigabit Ethernet interfaces. Whale asked us to use 100-Mbit/s interfaces, even though the e-Gap supports Gigabit Ethernet. PortWise says its mVPN, as a software product, works with any interface that can send TCP/IP traffic.

In most cases, network designers build SSL VPNs with just two interfaces – one to the outside world and one for internal traffic. However, several gateways support more than two interfaces. For example, Aventail supports clustering of its gateways; in this configuration, a third interface carries management traffic. NetScaler’s 9500 offers four interfaces for high availability. The Nortel product we tested had two interfaces; but another Nortel device, the Application Switch 2424-SSL, has 28 interfaces. The Symantec and Whale gateways also can support more than two interfaces if desired.

Going the other way, most gateways don’t even need two interfaces. All gateways tested except Whale’s can operated in “one-armed mode,” meaning traffic can enter and exit through a single interface.

Reverse Proxy

It’s a bit of an overstatement to say SSL gateways enable truly “clientless” access, but they do offer lots of choices of client types.

At the simplest level, an SSL VPN can be set up between a gateway and virtually any browser with SSL support. In this “reverse proxy” mode, a client outside the security perimeter requests some resource from the gateway. The gateway, in turn, sets up a new request to retrieve the desired resource (hence the term “proxy”).

Reverse-proxy mode has the advantage of ubiquity – it will work with any SSL-enabled browser from any operating system. This is a plus for applications that involve a wide mix of client machines, or machines running esoteric operating systems (think PDAs).

In terms of security, reverse proxies also have the benefit of hiding an organization’s internal structure for uniform resource locators (URLs). Since all client requests go to the external interface of the gateway, the clients don’t “see” the internal addresses or filenames. The downside is that external-to-internal mappings reside on the gateway; the more mapping that takes place, the more information must be stored. Further, to reach the same resource, users may have to request different URLs depending on whether they’re on the outside or inside of the gateway.

All products tested except NetScaler’s 9500 support reverse-proxy mode for SSL VPNs (see Table 3). NetScaler’s gateway supports reverse-proxy mode, but without authentication, so it can’t be used to set up VPNs in this mode. We tested all gateways except NetScaler’s in reverse-proxy mode.

Table 3: Access Mode Options

Reverse-proxy mode

Agent mode

ActiveX agents

Java agents

Array

X

X

(1)

X

Aventail

X

X

X

NetScreen (Neoteris)

X

X

X

X

NetScaler

(2)

X

X

(3)

Nortel

X

X

X

PortWise

X

X

X

Symantec (SafeWeb)

X

X

X

X

Whale

X

X

X

(1) ActiveX support was in field trials at test time
(2) Reverse-proxy mode does not support VPN authentication
(3) Java agent support scheduled for December 2003 release



Secret Agents

In another common mode of SSL VPN operation, the client machine authenticates with the gateway and then downloads a small piece of software, called an agent. Unbeknownst to the client machine, it will then make requests of the agent, which in turn securely retrieves the desired objects through the gateway.

Agent-mode operation, supported by all gateways in this test, has a couple of key advantages. First and foremost is customization. Commands can be embedded into URLs sent by client-side agents. By putting the right commands into an agent, it can launch virtually any application and tunnel to the client over an SSL connection. Since many organizations use custom applications to carry mission-critical (read: revenue-generating) traffic, these agents make it possible to run the applications remotely over SSL.

A second advantage with agents is that they hide complexity from the end-user. In reverse proxy mode, users may need to enter different URLs depending on which side of the gateway they’re on. Not so with agent-mode operation: The client sends the same URL regardless of location.

Agent types differ among vendors. At the time we tested (from September through November 2003), gateways from NetScaler, NetScreen, Symantec, and Whale supported agents based on ActiveX, Microsoft’s technology for integrating various controls into Web browsers. Array had ActiveX controls in field trials at test time, although we did not test them. Obviously, ActiveX agents work with Microsoft browsers, including those for various versions of Windows and the PocketPC operating system on some PDAs.

More universal support comes in the form of Java agents, supported in all gateways except NetScaler’s and Whale’s. NetScaler says it plans to ship a Java agent in December 2003.

Because of its platform independence, Java agents run on virtually any browser and operating system combination that supports the Java runtime engine (JRE). The list goes well beyond Windows to include the Apple Macintosh; virtually all versions of Unix, both commercial and open-source; and most handheld devices, including some mobile phones.

Light Reading Inc.’s market research organization, Heavy Reading, is publishing a complete list of all supported browsers and operating systems for SSL VPN gateways running in agent mode.

In our lab tests we measured performance in five different ways:

  • Session rate

  • Concurrent user capacity

  • Outlook Web access handling

    • – With denial-of-service attacks
      – Without denial-of-service attacks

  • Forwarding rate

We also assessed device security. (See Test Methodology for more details.)

We measured gateway scaleability with separate tests of rate and capacity. Rate describes how many SSL sessions a system can set up and tear down per second. Capacity describes the maximum number of concurrent users a system can handle.

In the session rate tests, we configured Spirent’s Avalanche client emulator/analyzer to log in to a gateway, request a 1-kbyte object from the Spirent Reflector server emulator, and log out. There’s no standard term yet to describe this set of SSL transactions, the way there is for, say, TCP connections. For simplicity’s sake we called the entire exchange an SSL “session.”

We configured Avalanche to keep adding more and more sessions until at least one transaction failed, and noted the highest rate at which the gateway processed transactions with no failures.

In presenting results, we’ve separated products into server-based and appliance-based categories. Among server-based entrants, PortWise’s mVPN posted by far the highest session rates, handling 155 sessions per second (see Figure 1).

44442_1a.gifPortWise’s strong showing is impressive, considering it’s a software-only product running on Windows (not famed as the fastest of operating systems). PortWise helped its cause considerably by supplying some very studly hardware: an HP Proliant DL360 with a single 3.06-GHz CPU and 3 Gbytes of RAM.

However, readers shouldn’t assume that fast server hardware alone makes a difference. Three other entrants – Aventail, Nortel, and Symantec – all supplied systems build on roughly comparable Dell Poweredge 1650 servers. There’s an order-of-magnitude difference between the Nortel and Symantec numbers, with Aventail about in the middle.

Nortel’s advantage makes sense, considering the vendor’s use of dedicated SSL acceleration hardware. As for Aventail and Symantec, there’s clearly a difference in the level of software optimization. Again, Symantec says it’s bringing out a newer, faster product in January.

NetScaler’s NS 9500 topped the appliance-based systems with rates of 1,100 sessions per second, followed by 850 sessions per second for Array, and 35 sessions per second for NetScreen (see Figure 2).

44442_2a.gif NetScaler’s first-place achievement in this event suggests its box can handle heavy loads on huge networks. Without taking anything away from NetScaler’s result, it’s important to put these numbers in perspective. The metric in this test is sessions per second. Multiplied out over time, the rates achieved by many (and arguably all) the systems tested will be more than sufficient for all but the largest enterprise networks.

Let’s illustrate the point with the lowest-rate result in this test: 8 sessions per second for the Symantec system. That works out to 480 users logging in and logging out per minute, or 28,800 users per hour. In the context of remote access, these numbers describe very high session rates indeed.

At the same time, remote access isn’t the only application for SSL VPNs. Consider the case of an enterprise that wants to use SSL authentication and encryption for all Web traffic, regardless of source. Applying the same math to NetScaler’s system, it would be possible to achieve 3.96 million sessions per hour. There are large enterprises that carry this many sessions, though few if any route all their traffic through a single box. If they wanted to do so, for some perverse reason, NetScaler’s gateway would be up to the task.

As the name suggests, the maximum concurrent users test measures how many users can do actual work through an SSL VPN gateway at a given instant.

In some ways, absolute capacity is an even more important measure of scaleability than maximum rate. As noted, distinctions between rates in the hundreds or thousands may not be meaningful for all but the very largest sites. In contrast, it’s a first-order priority to verify that any SSL gateway can scale up to support at least as many users as its software license permits.

To measure concurrent users, we configured Spirent’s Avalanche to log in many users, each requesting two 1-kbyte objects. Spirent’s Reflector returned the first object immediately.

The second object was special. Upon receipt of a request, Reflector waited 60 seconds before returning the object, thus holding the connection open. This 60-second delay had two purposes: First, it ensured existing clients’ connections didn’t go away as we added new users. Second, it forced each SSL gateway to maintain information on an ever-increasing number of users. We’re not claiming that 60-second-long Web transactions are commonplace; it’s just the method we used to ensure the gateways would hold open users’ connections.

We kept adding users during a five-minute ramp-up phase, and then observed each system handling transactions during a two-minute steady-state phase. As with the first test, we deemed a test run invalid if there were any unsuccessful transactions prior to the end of the steady-state phase.

Nortel’s SSL Accelerator led the server-based systems, allowing 2,500 users to work concurrently (see Figure 3). Following the Nortel gateway were systems from Aventail (1,195 users); Whale (1,000 users); PortWise (600 users); and Symantec (500 users).

44442_3a.gifThe appliance systems operated on a whole different scale (see Figure 4). NetScaler’s NS 9500 led the way, handling 58,000 concurrent users – the highest load of any system in this test. The NS 9500 may be able to scale even higher; the system’s nominal limit is 65,535 users, and the result we achieved reflected a limit of our test equipment.

44442_4a.gifFollowing in the appliance group were Array’s SP, with 16,000 concurrent users, and NetScreen’s Access 5000, with 1,367 users.

NetScreen’s result merits further explanation. The company says its A5000 supports 2,500 concurrent logins – the limit of its software license – but not necessarily 2,500 users concurrently retrieving Web objects. As evidence, the company cited customers that routinely max out their user licenses.

We don’t dispute NetScreen’s claim. At the same time, we think there’s a bit of hair-splitting going on here. In our view, a test of concurrent users should count the people actually using the system.

Another point to bear in mind is that not all SSL-enabled applications use short-lived TCP connections the way Web traffic often does. Some widely used remote access products, such as the Citrix Metaframe applications, keep TCP connections open as long as a user is logged in. These types of applications absolutely require concurrent logins and concurrent users to be one and the same.

If there’s a killer app for SSL-based VPNs, it is Outlook Web Access (OWA), the Web-based version of Microsoft Corp.’s popular Outlook mail client. Virtually all SSL VPN buyers we spoke with put OWA at the top of their lists for applications that benefit from anywhere/anytime access secured by SSL.

OWA builds complex Web pages on the fly. For example, displaying an inbox requires the transfer of nearly 80 objects – and that’s before opening or composing email. All these small-object transfers and the requisite rewriting of URLs require substantial processing power.

To test how well SSL gateways handle the load, we captured a real OWA session in which a user logs in, displays an inbox, and logs out. Using Avalanche, we played back the session with many users. We measured OWA performance with four metrics: maximum number of concurrent users without failures; transaction rate; and average URL and page response times.

As in earlier tests, we kept adding users until we found the maximum number the system could handle with no failed transactions. The results can charitably be described as variable.

Among server-based products, the PortWise gateway dwarfed its competitors by around two orders of magnitude and even gave appliance-based systems a run for the money (see Figure 5).

44442_5a.gifPortWise’s mVPN sustained requests from more than 20,000 users both with compression enabled and disabled; the next closest gateway, from Whale Communications, sustained a maximum load of 290 users without errors. (However, the PortWise gateway’s high user count came back to haunt it in other measures of OWA performance, as we’ll see shortly.)

Whale noted that its customers routinely run more than 290 users running OWA through its gateways. We have no reason to doubt that and we have no desire to inflict on Whale’s sales reps the pain of having to explain for the next few quarters why we got 290 instead of some much larger number.

There are two major differences between Whale’s production loads and our test setup. First, some of Whale’s customers use hardware-based SSL acceleration, but the company supplied a system without acceleration for testing. Customers doing acceleration in hardware should achieve substantially higher user counts than those we measured.

Second, Whale’s customers aren’t all asking for their OWA inboxes at the same time. If any more than 290 did so (assuming a software-based system), they’d see at least some errors, just as we did. Our tests did not attempt to model a “typical” user load (whatever that is). Rather, our goal was to find the limits of each system we tested. In the case of Whale’s e-Gap, 290 is it.

Among appliance-based gateways, NetScaler’s gateway easily outpaced the competition (see Figure 6). The NetScaler 9500 handled a load of 20,000 concurrent users, far more than 1,124 for Array, or 888 for NetScreen. As with Whale, NetScreen’s representatives noted that many customers routinely have 2,500 users concurrently running OWA. Again, we believe them – but NetScreen’s customers aren’t all getting their inboxes at the same time.

44442_6a.gifJust getting lots of users running OWA is far from the whole game. The real measure of OWA scaleability is whether the Nth user will get the same level of service as the first one. In many cases, our results suggest the answer is no.

A perfect result in our OWA tests would be large numbers of users and high transaction rates and low response times. That’s generally not what we saw.

PortWise’s mVPN may have set records in the concurrent user department, but each user operated far more slowly than the other server-based systems (see Figure 7). Nortel’s gateway had the by far highest OWA transaction rate, returning nearly 72 objects per second to each of 50 users. In contrast, the PortWise system returned 0.001 object per second to each of 20,000 users.

44442_7a.gifPortWise says its products with hardware-based SSL acceleration run faster than the software-based system it supplied to us for testing. Only the Aventail and Nortel systems delivered more than 1 object per second under load, in Nortel’s case thanks to its use of hardware-based SSL acceleration.

Among appliance-based products, the NetScaler 9500 came closest to our ideal of delivering high transaction rates to large numbers of users (see Figure 8). However, we should note that none of the transaction rates for appliances are particularly high. 44442_8a.gif

When calculated on a per-user basis, the fastest of the systems – Array’s Secure Proxy – handled around 5 transactions per second. We didn’t measure the OWA transaction rate for a single user, but we’d be willing to wager it would be far higher than 5 transactions completed per second. Our results suggest that ramping up the user count has a significant negative impact on transaction rates.

A related measurement of end-user satisfaction is response time. Response time measures the interval between the time users request something and the time they actually get it. We measured average response times for both individual URLs (the various objects in a page) and pages (all the objects that collectively make up a Web page). When it comes to response-time measurements, lower numbers are better.

The response-time measurements turned up more differences among products than any other test in this project. Among server-based gateways, Nortel’s gateway had by far the lowest response times. On average, the Nortel device returned objects in 20 milliseconds and pages in 341 ms. Then again, it also served the lowest number of concurrent users (see Figure 9).

44442_9b.gifAt the other end of the spectrum, the PortWise gateway held up average page response times by more than 2 minutes. This is the tradeoff when a software-based product runs on a platform with lots of memory (3 Gbytes, as supplied by PortWise) but no hardware assistance for SSL.

Array’s gateway turned in the lowest response times of the appliances (see Figure 10). The NetScaler gateway’s response times were substantially higher than Array’s; then again, it handled nearly 18 times as many users. However, the Array SP’s response times were lower than those of NetScreen’s A5000, even though the Array box handled more users. Again, the ideal box should handle large numbers of users, with low response times for all. In the end, there shouldn’t be any correlation between user counts and response times.

44442_10a.gifGiven that these are security products we tested, we also wanted to determine what impact, if any, we’d see on performance if the gateways were under attack.

To find out, we reran the OWA tests but this time configured Spirent’s Avalanche to launch four common forms of denial-of-service attacks: Smurf, SYN floods, UDP floods, and Xmas tree attacks.

We launched the DOS attacks while at the same time offering the exact same traffic we used in the OWA tests. Our goal was to determine what changes, if any, we’d see in gateway performance when the gateways were under attack.

Among server-based systems, the good news is that response times were generally about the same under attack as with no attack (see Figure 11).

44442_11a.gifHowever, gateways from Aventail, PortWise, and Symantec all failed to process at least one transaction when we added attack traffic into the mix. The Aventail system caused one or more transaction failures during all four of the attacks we launched, while the PortWise and Symantec products both failed while under SYN flood attack. PortWise’s gateway also had trouble with Xmas tree attacks.

There was a bit more variation in response times among appliance-based gateways under attack, perhaps because these systems handled higher rates to begin with (see Figure 12). Array’s system caused transaction failures when we mounted Smurf, UDP flood, and Xmas tree attacks.

44442_12a.gifArray says the transaction failures (which amounted to only a tiny fraction of the total) occurred because its system was already at its maximum capacity handling OWA traffic. That’s a fair point; in future tests we may modify this test to reduce the OWA load slightly from the maximum.

On the other hand, most of the server-based systems and both of the other appliance-based systems handled the attacks at their maximum load with no failed transactions. In Array’s case, the maximum load meant the system really couldn’t do any more work; for other systems, there were still cycles available to fend off attacks.

We’ll skip the comparisons of URL and page response times with and without attacks, because results were similar to those for transaction rates. On the plus side, response times under attack were generally no worse than with no attack. On the minus side, we observed a small number of failed transactions when we launched attacks against gateways from Array, Aventail, PortWise, and Symantec.

Our forwarding rate tests sought to answer a key question of any network device: how quickly it moves data. In the case of SSL VPNs, the task is complicated by the compute-intensive tasks of encrypting and decrypting traffic and rewriting URLs.

Just to make things even more stressful (stress is a good thing in device testing), we took the same approach in this event of measuring with the greatest number of users each system could handle with no failed transactions such as timeouts (see Figure 13). These counts were all over the map, ranging from 500 users for Array to just 1 for Whale.

44442_13a.gifThere are two points to bear in mind regarding the user counts from the forwarding rate tests. First, as a rule the systems can support far greater numbers of users than the numbers we observed in this test – but not without errors when handling this specific load.

In the most extreme example, Whale’s e-Gap supported 1,000 users in the concurrent user test, and just one user here. The difference is explained by the traffic load involved. When moving data at high rates (the goal of this test) we observed errors with Whale’s e-Gap in tests with even two users. The previous tests, in which Whale’s e-Gap handled a load of 1,000 concurrent users, involved the transfer of a trivial amount of data – only 1 kbyte per user every 60 seconds.

Second, the forwarding rates we measured aren’t necessarily the highest each system can achieve. Rather, they represent the most each system can pump out while handling the maximum number of users.

For example, a system might achieve a rate of 300 Mbit/s with 235 users, but dip to 290 Mbit/s with 250 users. In such cases we went with the highest user count even though it didn’t necessarily reflect the highest forwarding rate.

Among server-based systems, PortWise’s gateway ran by far the fastest, moving traffic at 310 Mbit/s both with and without compression enabled (see Figure 14). Compression didn’t make any difference in PortWise’s results because it took more time to compress and decompress the large object we used than it did to transfer the (somewhat) smaller compressed object.

44442_14a.gifAt the other end of the spectrum were the gateways from Symantec and Whale, which moved traffic at 4 Mbit/s and less than 1 Mbit/s, respectively. Both companies supplied systems with 100-Mbit/s fast Ethernet interfaces (all others used Gigabit Ethernet copper interfaces) and no hardware-based SSL acceleration. Symantec says its new product will deliver much higher performance.

Whale says it already has other products with SSL acceleration hardware, and these deliver significantly higher forwarding rates. We have no reason to dispute that claim. At the same time, we should note that other products without hardware acceleration ran far faster than the Symantec or Whale gateways. The PortWise gateway – which turned in the fastest rates among server-based products – was one of these.

Among appliance-based products, NetScaler’s gateway edged out Array’s with the highest rate in the test – nearly 500 Mbit/s (see Figure 15). NetScreen’s A5000 trailed the other two appliances with rates of 249 Mbit/s.

44442_15a.gifWhile there was only a small difference between forwarding rates for the NetScaler and Array gateways, it’s important to note that Array achieved its result with 500 users, while NetScaler did so with just 12. We ran tests with a higher number of users on the NetScaler box, but results were invalid, since these involved one or more transaction failures. NetScaler’s engineers told us we’d hit the limit of what the company’s gateway could deliver.

In the 1994 prison drama The Shawshank Redemption, the guards spring a surprise inspection on inmate Tim Robbins’s cell but find little of concern: “Some contraband here, but nothing to get in a twist over.” The same could be said of our security testing of the SSL gateways.

We subjected each gateway to a thorough scan using Nessus, an open-source vulnerability assessment tool. Even though Nessus checks for more than 1,700 known vulnerabilities, the issues it turned up were minor in nature (see Table 4).

Table 4: Security Assessment

Array

Aventail

Net Screen

Net Scaler

Nortel

Port Wise

Symantec

Whale

SSLv2 connections accepted

X

X

X

X

SSLv2 offers weak ciphers

X

X

X

X

Does not discard SYN packets with FIN flag set

X

X

X

X

X

X

SSHv1 connections accepted

X



Nessus did report a few alarming-sounding vulnerabilities, but in every case we determined these were false positives. We’re not presenting those results here.

In terms of SSL security, the gateways from NetScreen, PortWise, Symantec, and Whale all accept connections from browsers using SSLv2, which has known cryptographic weaknesses. As a result, traffic sent over connections using SSLv2 could be intercepted and decrypted with relative ease. Because of these vulnerabilities, it’s generally considered a good security practice to disable SSLv2 access.

It should be pointed out that most Web browsers – including Microsoft’s market-dominating Internet Explorer – support SSLv2 by default. Still, the SSL version used for a connection is something the gateway can control. Users can, and should, disable SSLv2 support before putting these systems into production.

Gateways that supported SSLv2 connections also generally allowed a range of encryption algorithms, or ciphers, including some considered vulnerable to brute-force attacks. Since the gateways also offer much stronger ciphers and use them by default, the availability of weak ciphers doesn’t necessarily mean data can be compromised.

The weaker ciphers are sometimes called export-class ciphers because many countries don’t allow the importation of certain strong ciphers. Weak ciphers may help service clients in those countries, but, as with SSLv2, the most secure configuration option would be to disable them.

Nessus found that all products except Array’s and Symantec’s responded to TCP SYN packets (those used to open a connection) that also had a FIN flag set (indicating that the gateway should close, not open, a connection). Not discarding SYN/FIN packets requires a bit of extra work on the part of the gateway, but in no case were we able to proceed and actually complete a connection started with one of these bogus packets.

Aventail’s gateway supports remote management via Secure Shell (SSH) version 1, which has known cryptographic vulnerabilities, and version 2, which does not. Since the gateway will offer SSHv2 first by default, traffic from any client that supports SSHv2 will not be vulnerable. However, not all clients support SSHv2. It is generally considered good security etiquette to disable SSHv1 access.

Thanks

Light Reading gratefully acknowledges the support of vendors who made it possible to conduct its largest-ever test.

Spirent Communications supplied not only its Avalanche and Reflector traffic generator/analyzers but also furnished considerable engineering support. Jerry Perser drafted the performance test methodology, while Philip Joung and Timmons Player helped conduct tests and interpret results. The following Spirent employees also offered logistical assistance: Bob Anderson, Neil Anderson, Roy Chua, Brooks Hickman, Rob Johnson, Michelle Rhines, Angus Robertson, and Bernardo de los Santos.

Thanks also to Foundry Networks Inc., which supplied its FastIron Edge 12GCF copper Gigabit Ethernet switch as test bed infrastructure. The 10/100/1000 interfaces of the Foundry switch tied everything together and didn’t drop a single packet in nearly three months of grueling tests.

Finally, thanks to the many developers of Nessus, an open-source vulnerability assessment tool.

Subscribe and receive the latest news from the industry.
Join 62,000+ members. Yes it's completely free.

You May Also Like