Shocking Results in LR's SSL VPN Test

Light Reading has completed its biggest ever technology test, with eight (8!) SSL-based VPN gateways put through their paces. And the findings will be like gold dust to any network manager with even a passing interest in VPN technology (see SSL VPNs: Access Anywhere, Anytime ).

Our testing partners, Spirent Communications and Network Test, got their hands on the leading products from the following vendors:

The notable absentee from that list is Cisco Systems Inc. (Nasdaq: CSCO), which only entered the market as the tests were nearly complete (see Cisco Takes a Stab at SSL).

The outcome? A mixed bag. Each of the vendors' gateways has a core strength, but there is no clear winner in terms of overall functionality, and there is still plenty of work to be done to improve each of the products.

Certain products excelled in individual tests, and the report offers a crucial breakdown of which products came out on top in each area, along with an extensive analysis of the results.

Aventail, for example, offers the broadest range of configuration options, while NetScreen's user interface was commended for its clarity. NetScaler set the pace in scaleability, but Nortel performed exceptionally well in the tests of server-based systems.

The gateways were subjected to a range of performance and scaleability tests to determine which products performed best in terms of: session rate; concurrent user capacity; Microsoft Outlook Web Access (OWA) handling (with and without denial-of-service attacks); forwarding rate; and security.

The overall findings of the test are:

  • Secure Sockets Layer VPNs are generally scaleable, handling from hundreds up to tens of thousands of concurrent users per gateway, with NetScaler’s NS 9500, which supports up to 58,000 users at one time, top of the pile.

  • While all the tested systems support OWA, there’s lots of variation in how well they do so, with some response times slowing to minutes when there are a lot of concurrent users.

  • The best performers were those with hardware-based SSL acceleration, with some pushing data at an impressive 500 Mbit/s, while those performing encryption/decryption tasks in software ran as slowly as 1 Mbit/s.

  • The products tested were generally secure, with only a few wrinkles, mainly involving weak default configurations, uncovered.

    The results, according to David Newman, president of Network Test, provide vital guidance to users because of the unique methodology deployed. "We hope our tests help enterprise network architects make informed decisions in terms of features, performance, and security,” says Newman. He says the test attracted a lot of participants because "we created a new methodology, one that includes what is arguably the killer app for remote access: OWA, the Webified version of Microsoft Outlook."

    And it's a methodology that has legs. "This is a milestone test in that it establishes an industry standard methodology for testing SSL VPNs,” says Mark Fishburn, VP of technical strategy at Spirent Communications, who describes the test as "the most comprehensive SSL VPN test ever performed in this competitive market segment."

    And why are these products important? Because SSL VPNs, which enable access from almost any Web browser, look to offer a very neat complement to IPSec VPNs, as they are very well suited for remote access and extranet applications, which have been sore spots for IPSec.

    — Ray Le Maistre, International Editor, Boardwatch

  • paulfee 12/4/2012 | 11:11:06 PM
    re: Shocking Results in LR's SSL VPN Test Hello all,

    Can someone tell me how these SSL VPN products avoid the TCP over TCP meltdown problem?

    Using SSL for VPNs, one can get a protocol stack as follows:

    1) Application (e.g. Web browser)
    1) HTTP
    1) TCP (upper)
    1) IP
    2) SSL
    2) TCP (lower)
    2) IP
    2) Ethernet

    1 = Layers generated by end user (e.g. Web browser on PC).
    2 = Layers generated by VPN box.

    Use of TCP over TCP can cause a meltdown effect. If the lower TCP connection (that carrying SSL) has slower timers (e.g. due to congestion) then the upper TCP (that carrying the application) may try to retransmit faster than the lower TCP.

    The upper TCP expects to be delivered over an unreliable network, however the lower TCP is already guaranteeing reliability over the VPN.

    The upper TCP may reside on a node other than the VPN box (e.g. a normal PC), therefore it can't be modified without inconvenience to the user.

    The lower TCP resides on the VPN box and is a candidate for modification. Have the VPN vendors modified their TCP implementation in some way to avoid TCP over TCP meltdown?

    Without addressing this issue a VPN may operate normally under uncongested conditions, then rapidly degrade when congestion is experienced. Some products simply reset the SSL connection on a regular basis to avoid this.

    Ideally the VPN should be transported over an unreliable datagram protocol, however SSL uses TCP. Use of IP (e.g. IPSec) solves the problem but is not as firewall friendly. Use of UDP would be better, but is not compatible with SSL.

    I'd like to read your comments.

    Paul Fee
    [email protected]__remove_this_bit__.com
    dnewman 12/4/2012 | 11:11:00 PM
    re: Shocking Results in LR's SSL VPN Test "Can someone tell me how these SSL VPN products avoid the TCP over TCP meltdown problem?"

    Hi Paul,

    Great question. Multiple folks in this test (Array and Netscaler for sure, there may be others) multiplex TCP connections to the server, which helps mitigate the problem you describe.

    "Hhe lower TCP resides on the VPN box and is a candidate for modification. Have the VPN vendors modified their TCP implementation?"

    Short answer is yes. Lots of products in this area may run on Linux/FreeBSD/OpenBSD, but they've hacked both the TCP stack and written their own application-layer proxies so that the performance will be very different than a vanilla install of one of those OSs.

    The real proof is in the numbers from the test. Forget about device internals for a moment, and ask the question of how these devices perform when handling lots of traffic from lots of users. As you can see, there were many cases in our tests where a large number of users led to large response times and reduced transaction rates.

    David Newman
    Network Test

    DISCLAIMER: Network Test is neither long nor short in any of the companies mentioned in this article.

    Sign In