5:05 PM -- I have spent a lot of time running benchmarks on wireless LANs, but almost always using small configurations. I once ran a test using 10 APs and 50 clients; just finding a suitable building to house all of this was a challenge. It took about three days to set everything up, verify the configurations, and get the results. We used IxChariot for running the tests, which is not inexpensive, easy to use, or optimized for testing wireless LANs. And, of course, we got results only for one static configuration in one venue with whatever RF interference was present at the time of the test.

This experience led to an idea -- could we utilize electronic test equipment, of the type used by product engineers, to run large-scale benchmark tests in an isolated, repeatable environment? And would the results of such tests be indicative of what we would see in the real world?

Well, I finally did the first series of tests along these lines, and the results can be seen in the Farpoint Group Technical Note available here. We tested Aruba Networks Inc. (Nasdaq: ARUN) vs Meru Networks Inc. in a number of dimensions, and the real-world results showed an excellent correlation to those obtained from the test equipment, in this case a VeriWave Inc. WaveTest 90. While this first study is hardly definitive, I’m encouraged enough to dub the concept of testing in isolation virtual benchmarking. And I think enterprises will eventually use this technique for head-to-head evaluations in place of real-world testing. Sure, the equipment involved needs to be useable by mere mortals, and the RF channel modeling needs to be more robust, especially as MIMO and .11n take center stage. But the potential savings in time and hard currency is more than inviting.

And, in case you’re wondering, Aruba kicked butt in the tests. Those guys continue to amaze me.

— Craig Mathias is Principal Analyst at the Farpoint Group , an advisory firm specializing in wireless communications and mobile computing. Special to Unstrung