What Analysts Do

3:15 PM -- Some of the feedback I’ve received, both directly and indirectly, from some of the firms whose products we evaluated in our “draft n” benchmark tests has motivated me to state what should be obvious, but, sadly, is not always today. Analysts like me are experts in a particular area of technology; we live and breathe our particular subjects with a passion that’s hard to explain to anyone who’s not a total nerd (a nerd, by the way, is a geek with at least some social skills; that’s me). We’re experts because we’ve made every conceivable mistake in our chosen areas and learned at least enough not to make them again.

The tests I recently ran illustrated, to me, anyway, that the so-called “draft n” 802.11 products I tested didn’t perform as well as at least one non-draft-complaint MIMO-based product that’s been on the market for a while. (See Draft What?) They also gave me the opportunity to rant once again against the whole “draft n” concept, which I consider to be misleading at best and otherwise just plain wrong. I think I’ve explained my reasoning in the statements in the report, and I won’t cover all that again here. Anyway, that’s analysis.

But I do, sadly, need to state here unequivocally that the opinions in anything I publish are mine and mine alone. No ethical analyst (read: no analyst worthy of being one) would tailor results from experiments in any way that deviates from the reality revealed in a given benchmark or other quantifiable exercise (i.e., the truth). I occasionally read about scientific fraud in a variety of fields. One example of years ago hit very close to home. A doctor who treated me in high school, a world-famous dermatologist, claimed to have solved the rejection problem in skin transplants, using mice, anyway. What he did, though, was to paint a black patch on a white mouse using a magic marker. Pretty stupid, huh? Especially for an otherwise smart guy (I hope so, anyway, said the patient).

As you’ve seen, I carefully document the test conditions and results of any benchmark project, and use techniques like turntables and spectrum analyzers that other people usually don’t. I welcome criticism of the process and the results, and will provide assistance at no charge to anyone attempting similar tests. But I will not put up with sour-grapes criticism of my motivations or ethics, at least not without proof. No ethical person would.

— Craig Mathias is Principal Analyst at the Farpoint Group , an advisory firm specializing in wireless communications and mobile computing. Special to Unstrung