Forecasting Follies
It’s not a great year to be in the forecasting business, to be sure.
The discussions on the message boards following recent articles on leading research firms’ revisions have sparked plenty of rancor, with some questioning the usefulness of forecasts at all, good times or bad. Never is a market forecaster more challenged than when a market is in free fall. If a market that has been increasing unabated for a decade suddenly diminishes by over 30 percent, how could forecasters have missed it?
The quick answers are that these forecasters are little more than shills for the industry; they’re all so young that they’ve never lived through a down market and thus couldn’t see the signs coming; and/or their customers are often startups that are influencing forecasts by offering all manner of perks.
Well, some of that must certainly be true, but there is much more to a market research report than forecasts, and the business of market research is just that: a business. Regardless of the stance any research firm takes, these are not non-profits serving the public interest. These are businesses that are under pressure to produce profits, sell reports, add clients, put people in chairs at conferences, and renew subscriptions to newsletters.
The difficult fact every market research firm must face at some time is that good news sells better than bad, and a researcher is often under pressure to put the glossiest sheen on an analysis – or at least avoid talking about markets that are on their way down and focus on those with a strong future. Optical was thought to be one of those, and now these firms are finding they were wrong. So now it’s time to look again at how they position themselves in a down market.
Some will take the high road and claim they don’t share the biases of the analyst firms that “sold out” to startup vendors; others will just focus on markets that still show some growth so their forecasts can remain upward; while others, such as RHK, have to pull in their horns and put out their revisions – then take their lumps and get on with the work of acquiring a deeper understanding of this volatile and unpredictable market.
I’ve written more than 30 reports in ten years, most on optical networks markets, most with rosy forecasts – some accurate, some way off. Now that I’m out of the forecasting business for the time being, I can share some insight into how a report is built and how such seemingly egregious errors in judgment can be made, given how close to the market a researcher is over the three or four months he or she spends composing one of these works.
I should note here that the following applies to my own experience, which has been limited to the kind of report often called a “market and technology assessment,” one that includes a detailed analysis of an entire market – the technological trends underpinning it, the market forces at work, and the outlook for that market over a five-year period. (Once upon a time, ten-year forecasts were common, but those were thrown out in the mid 90s when the Internet made predicting beyond a few years an act of hubris, not reason.)
There are typically five stages to authoring a report: (1) picking a topic, (2) talking to the players, (3) understanding the technology, (4) writing the damn thing, and (5) building the forecasts. At Pioneer Consulting, my analysts and I all followed this model, and we put out some large reports: Most came in at over 350 pages, guided by the notion that in a market driven by new technology adoption, the bigger the report the better. Why?
Since we were typically looking at markets that didn’t yet exist, the main task of a researcher was to just describe and define that market as precisely as possible. The forecasts, in these cases, were not nearly as important as the rest of the report, because when these were put together, not many folks in the industry understood the topic at all – so what became most valuable to a customer was not just how big the market might be (which they all suspected was pure speculation) but just what this market was and what was needed to win in it.
A good report to look at as an example is Pioneer Consulting’s “Optical Switching Systems,” which I authored in the second half of 1999, after I had completed a metro optical networks report.
Step one, picking the topic, was easy. When we picked a topic for a report we had a few hard and fast criteria: Is the topic about to become hot in the next six months? Is there a market for this product now? (“No” was preferable in the latter case.) Is there startup activity around this topic – meaning, is there private capital fueling the creation of new vendors and challenging the incumbent vendors? If yes, then many potential customers exist for the report and interest in the topic will likely be sustained over a long period of time.
I had talked with Charles Chi of Lightera (now part of Ciena) at OFC 98 and had a good sense of just what an optical switching system was. I had seen the MONET demonstration at an OFC also, and saw what Tellium was up to. And I had heard just enough about other companies (Lucent, Nortel, etc.) that it seemed evident that this topic was solid.
The next step was talking to the players, and it is worth noting here that startups do in fact engage consultants more actively than the “Big Five” do, plain and simple. They are hungry, anxious to get their message out, genuinely excited about the market opportunity, and well funded enough at the time to travel around and meet with analysts and talk shop. We had to hunt the other folks down, which we did and got a similar message: The rapid deployment of DWDM was creating a mess in core nodes, slowing provisioning of capacity, putting an invisible cap on scaleability, and in general making a mess out of optical network management.
Coming off the metro DWDM report, this was a breath of fresh air. There was little debate whether the market would take off, it was just a matter of determining when and how big.
As for the technology, the vendors all had common positioning: Optical switching systems were utterly necessary, without reasonable alternatives. The only debate surrounded whether OEO switches would last in the network or give way to all-optical switches, a rather technical controversy that could not be answered satisfactorily at the time, because carriers hadn’t yet had a chance to evaluate either in any detail.
Carriers interviewed all liked the idea of optical switching. From the largest to the smallest, they were saying, “Yes, we need these,” and some were saying they needed them in large quantities with very high port counts to accommodate the immense growth in traffic they were witnessing on their networks. WorldCom, for one, went so far as to say they needed a 4000-port switch by 2004, with OC-768 ports! That is the kind of information that gets startups salivating. That, I would submit here, is the beginning of hype.
The notion that carriers are the “oracle” and through a series of interviews you can get “unbiased” information from them – versus the self-interested information one receives from vendors – is often not true. The odd thing about doing market research in late 1999 was that carriers were all caught off guard by the rise of the Internet, and most were simply reacting, not planning. When they did plan, their forecasts of traffic transiting their core nodes were fantastically high, exponential. Multiple terabits of connectivity would be required in just a few years, and this connectivity had to be managed, automated, controlled by intelligent network elements.
After talking with all the vendors in the space, and checking their claims with carriers, I came away with the distinct impression that optical switching was all but a foregone conclusion, and the market would be ramping quickly. This was around September 1999. An optical switching report is much easier to write today than it was then.
With all that information in hand, I set about writing the report, which took over three months. I prepared vendor profiles of startups and incumbents and carrier profiles that included maps of their network infrastructures and locations of the POPs that would most likely be candidates for optical switching systems. I wrote an analysis of bandwidth demand, the resulting deployment of DWDM, and the fundamentals of optical switching technology.
Writing is an interesting process. Oft times you don’t know what you know until you write it, and many times the process of writing reveals truths that weren’t apparent when you started out. Some (apparent) truths that became clear during the writing process: I began to believe that OEO switches (particularly the grooming kind) would dominate and ultimately prevail over all-optical switches; that MEMS could run into serious problems; and that grooming made sense, if it came at little cost penalty. This all comprised more than 300 pages of writing.
Next came the forecasts, which have become the bane of market research firms this year. You have to stick your neck out when you do these, particularly when you have no market currently in existence. This was always the challenge at Pioneer: forecasting in a vacuum. Because there was no historical data to draw on as a foundation, we employed a “penetration rate” method of forecasting.
The idea is straightforward. You have a new product that is being introduced into the public network, so you figure out how many could ultimately be deployed, and by weighing the enabling and limiting factors of the marketplace, then determine a forecast for how many will be deployed each year. For optical switching, we counted up the addressable POPs in North America and Europe, created a bandwidth demand forecast to determine the number of OC48s and OC192s terminating at those POPs, then forecast the price per port of optical switches. The number of circuits was derived from separate forecasts on Internet traffic demand (around 89% per year, by our estimations, which turned out to be rather accurate and not at all part of the hype of that year) and DWDM deployment trends.
There were plenty of limiting factors, including the ability of vendors to bring these products to trial and then to market, carrier willingness to adopt a substitute technology, and, of course, carriers' capex budgets. These turned out to be the devils in the research that would only show themselves later, once the report was published and in customers' hands.
With those basics in place we could start forecasting the market for optical switching systems. Our methodology produced a number that seemed awfully high at first, but then after some checking I found that it was reasonable as a “market opportunity” forecast, in essence saying: This is how big the market could be if all the assumptions are correct. The nice thing about this kind of forecasting is that it is based on a very evident methodology, a series of dials you can turn to a degree that matches the effect of the assumption. Tune the dials one way or the other and you get very different results.
It’s interesting that most people believe that startups (and their VCs) drove these high forecasts and exuberance about the market, but in fact it was a startup that first alerted me that a couple of the assumptions were off. They didn’t want inaccurate forecasts, since their own internal sales forecasts were derived in part from these. They needed forecasts to be as realistic as possible, so their goals could be realistic.
My startup informants noted the assumption that carriers would be taking fully outfitted 256-port OEO grooming switches was turning out not to be true. The carriers preferred to start small, burn them in over a long period of time, and complete network management integration. The second assumption that was off was the relationship between DWDM deployment and switch size. Carriers were starting to look more closely at how they would implement optical switches, and it didn’t often result in hooking all their DWDM channels up to a system: Typically it was only a fraction of those and only in a fraction of their POPs. This news was all coming in after the report was published, so the forecasts were having to be revised downward after the report went out.
That stinks, but it wasn’t that we were a victim of hype. Rather, we were victimized by ignorance – which could be attributed to all of the players, vendors and carriers alike. This is the main risk you take being the first to publish a report on a given topic. The risks other analysts run publishing numbers on a market that is already in full bloom is that you miss a downturn, or worse yet, a crash. You won't get let off the hook easily for that because your numbers are everything.
The market downturn eventually followed, which caught everyone off guard and was nearly impossible to predict in its magnitude. We talked to lenders and VCs through the years, and they were all saying they would continue to fund the growth of vendors and new service providers alike. The industry was in a period of transformation that would require completely new network operators, novel forms of equipment. There was some cognizance of the hype surrounding the optical market, but most folks patted themselves on the back, saying what happened to the dotcoms could never happen here because we had intellectual property, we had real hardware that solved a real problem that carriers would pay for. It was a comforting argument and kept most of us from acknowledging the inevitable once it started to show itself.
The key belief we clung to was that legacy equipment would suffer, but not next-gen equipment like optical switching. That belief was ultimately foiled by new operator debt, which prevented them from building out their next-gen networks as quickly as hoped. Once their dreams were dashed, so went the next-gen optical market, right down with the legacy market.
So are market researchers such frauds? Most of our customers responded well to the reports, taking the forecasts with the appropriate grain of salt, but found the detailed competitive analysis and technology assessment very useful and informative. The forecasts were helpful in raising money, to be sure, but they were also fungible enough to allow the user to make their own decisions about the assumptions and alter the outcome accordingly.
Writing large reports was good that way, you had a lot to offer the customer, not just a spreadsheet. Those companies that do very detailed quarter-to-quarter forecasting have the hardest task in terms of numbers, but in the end they are just numbers. Did they really contribute to the demise of this market? I doubt it. The frenzy was vivid enough that most market researchers were watching it with the same wide eyes as everyone else. Providing useful analysis doesn’t always involve getting forecasts perfect. Often it means characterizing a market opportunity clearly enough to empower the players with knowledge they can use to succeed in any climate.
Good luck to the folks forecasting optical networks markets in the next couple years. Their mettle will be tested.
— Scott Clavenna, Director of Research, Light Reading
http://www.lightreading.com
The discussions on the message boards following recent articles on leading research firms’ revisions have sparked plenty of rancor, with some questioning the usefulness of forecasts at all, good times or bad. Never is a market forecaster more challenged than when a market is in free fall. If a market that has been increasing unabated for a decade suddenly diminishes by over 30 percent, how could forecasters have missed it?
The quick answers are that these forecasters are little more than shills for the industry; they’re all so young that they’ve never lived through a down market and thus couldn’t see the signs coming; and/or their customers are often startups that are influencing forecasts by offering all manner of perks.
Well, some of that must certainly be true, but there is much more to a market research report than forecasts, and the business of market research is just that: a business. Regardless of the stance any research firm takes, these are not non-profits serving the public interest. These are businesses that are under pressure to produce profits, sell reports, add clients, put people in chairs at conferences, and renew subscriptions to newsletters.
The difficult fact every market research firm must face at some time is that good news sells better than bad, and a researcher is often under pressure to put the glossiest sheen on an analysis – or at least avoid talking about markets that are on their way down and focus on those with a strong future. Optical was thought to be one of those, and now these firms are finding they were wrong. So now it’s time to look again at how they position themselves in a down market.
Some will take the high road and claim they don’t share the biases of the analyst firms that “sold out” to startup vendors; others will just focus on markets that still show some growth so their forecasts can remain upward; while others, such as RHK, have to pull in their horns and put out their revisions – then take their lumps and get on with the work of acquiring a deeper understanding of this volatile and unpredictable market.
I’ve written more than 30 reports in ten years, most on optical networks markets, most with rosy forecasts – some accurate, some way off. Now that I’m out of the forecasting business for the time being, I can share some insight into how a report is built and how such seemingly egregious errors in judgment can be made, given how close to the market a researcher is over the three or four months he or she spends composing one of these works.
I should note here that the following applies to my own experience, which has been limited to the kind of report often called a “market and technology assessment,” one that includes a detailed analysis of an entire market – the technological trends underpinning it, the market forces at work, and the outlook for that market over a five-year period. (Once upon a time, ten-year forecasts were common, but those were thrown out in the mid 90s when the Internet made predicting beyond a few years an act of hubris, not reason.)
There are typically five stages to authoring a report: (1) picking a topic, (2) talking to the players, (3) understanding the technology, (4) writing the damn thing, and (5) building the forecasts. At Pioneer Consulting, my analysts and I all followed this model, and we put out some large reports: Most came in at over 350 pages, guided by the notion that in a market driven by new technology adoption, the bigger the report the better. Why?
Since we were typically looking at markets that didn’t yet exist, the main task of a researcher was to just describe and define that market as precisely as possible. The forecasts, in these cases, were not nearly as important as the rest of the report, because when these were put together, not many folks in the industry understood the topic at all – so what became most valuable to a customer was not just how big the market might be (which they all suspected was pure speculation) but just what this market was and what was needed to win in it.
A good report to look at as an example is Pioneer Consulting’s “Optical Switching Systems,” which I authored in the second half of 1999, after I had completed a metro optical networks report.
I had talked with Charles Chi of Lightera (now part of Ciena) at OFC 98 and had a good sense of just what an optical switching system was. I had seen the MONET demonstration at an OFC also, and saw what Tellium was up to. And I had heard just enough about other companies (Lucent, Nortel, etc.) that it seemed evident that this topic was solid.
Coming off the metro DWDM report, this was a breath of fresh air. There was little debate whether the market would take off, it was just a matter of determining when and how big.
Carriers interviewed all liked the idea of optical switching. From the largest to the smallest, they were saying, “Yes, we need these,” and some were saying they needed them in large quantities with very high port counts to accommodate the immense growth in traffic they were witnessing on their networks. WorldCom, for one, went so far as to say they needed a 4000-port switch by 2004, with OC-768 ports! That is the kind of information that gets startups salivating. That, I would submit here, is the beginning of hype.
The notion that carriers are the “oracle” and through a series of interviews you can get “unbiased” information from them – versus the self-interested information one receives from vendors – is often not true. The odd thing about doing market research in late 1999 was that carriers were all caught off guard by the rise of the Internet, and most were simply reacting, not planning. When they did plan, their forecasts of traffic transiting their core nodes were fantastically high, exponential. Multiple terabits of connectivity would be required in just a few years, and this connectivity had to be managed, automated, controlled by intelligent network elements.
After talking with all the vendors in the space, and checking their claims with carriers, I came away with the distinct impression that optical switching was all but a foregone conclusion, and the market would be ramping quickly. This was around September 1999. An optical switching report is much easier to write today than it was then.
Writing is an interesting process. Oft times you don’t know what you know until you write it, and many times the process of writing reveals truths that weren’t apparent when you started out. Some (apparent) truths that became clear during the writing process: I began to believe that OEO switches (particularly the grooming kind) would dominate and ultimately prevail over all-optical switches; that MEMS could run into serious problems; and that grooming made sense, if it came at little cost penalty. This all comprised more than 300 pages of writing.
The idea is straightforward. You have a new product that is being introduced into the public network, so you figure out how many could ultimately be deployed, and by weighing the enabling and limiting factors of the marketplace, then determine a forecast for how many will be deployed each year. For optical switching, we counted up the addressable POPs in North America and Europe, created a bandwidth demand forecast to determine the number of OC48s and OC192s terminating at those POPs, then forecast the price per port of optical switches. The number of circuits was derived from separate forecasts on Internet traffic demand (around 89% per year, by our estimations, which turned out to be rather accurate and not at all part of the hype of that year) and DWDM deployment trends.
There were plenty of limiting factors, including the ability of vendors to bring these products to trial and then to market, carrier willingness to adopt a substitute technology, and, of course, carriers' capex budgets. These turned out to be the devils in the research that would only show themselves later, once the report was published and in customers' hands.
With those basics in place we could start forecasting the market for optical switching systems. Our methodology produced a number that seemed awfully high at first, but then after some checking I found that it was reasonable as a “market opportunity” forecast, in essence saying: This is how big the market could be if all the assumptions are correct. The nice thing about this kind of forecasting is that it is based on a very evident methodology, a series of dials you can turn to a degree that matches the effect of the assumption. Tune the dials one way or the other and you get very different results.
It’s interesting that most people believe that startups (and their VCs) drove these high forecasts and exuberance about the market, but in fact it was a startup that first alerted me that a couple of the assumptions were off. They didn’t want inaccurate forecasts, since their own internal sales forecasts were derived in part from these. They needed forecasts to be as realistic as possible, so their goals could be realistic.
My startup informants noted the assumption that carriers would be taking fully outfitted 256-port OEO grooming switches was turning out not to be true. The carriers preferred to start small, burn them in over a long period of time, and complete network management integration. The second assumption that was off was the relationship between DWDM deployment and switch size. Carriers were starting to look more closely at how they would implement optical switches, and it didn’t often result in hooking all their DWDM channels up to a system: Typically it was only a fraction of those and only in a fraction of their POPs. This news was all coming in after the report was published, so the forecasts were having to be revised downward after the report went out.
That stinks, but it wasn’t that we were a victim of hype. Rather, we were victimized by ignorance – which could be attributed to all of the players, vendors and carriers alike. This is the main risk you take being the first to publish a report on a given topic. The risks other analysts run publishing numbers on a market that is already in full bloom is that you miss a downturn, or worse yet, a crash. You won't get let off the hook easily for that because your numbers are everything.
The market downturn eventually followed, which caught everyone off guard and was nearly impossible to predict in its magnitude. We talked to lenders and VCs through the years, and they were all saying they would continue to fund the growth of vendors and new service providers alike. The industry was in a period of transformation that would require completely new network operators, novel forms of equipment. There was some cognizance of the hype surrounding the optical market, but most folks patted themselves on the back, saying what happened to the dotcoms could never happen here because we had intellectual property, we had real hardware that solved a real problem that carriers would pay for. It was a comforting argument and kept most of us from acknowledging the inevitable once it started to show itself.
The key belief we clung to was that legacy equipment would suffer, but not next-gen equipment like optical switching. That belief was ultimately foiled by new operator debt, which prevented them from building out their next-gen networks as quickly as hoped. Once their dreams were dashed, so went the next-gen optical market, right down with the legacy market.
So are market researchers such frauds? Most of our customers responded well to the reports, taking the forecasts with the appropriate grain of salt, but found the detailed competitive analysis and technology assessment very useful and informative. The forecasts were helpful in raising money, to be sure, but they were also fungible enough to allow the user to make their own decisions about the assumptions and alter the outcome accordingly.
Writing large reports was good that way, you had a lot to offer the customer, not just a spreadsheet. Those companies that do very detailed quarter-to-quarter forecasting have the hardest task in terms of numbers, but in the end they are just numbers. Did they really contribute to the demise of this market? I doubt it. The frenzy was vivid enough that most market researchers were watching it with the same wide eyes as everyone else. Providing useful analysis doesn’t always involve getting forecasts perfect. Often it means characterizing a market opportunity clearly enough to empower the players with knowledge they can use to succeed in any climate.
Good luck to the folks forecasting optical networks markets in the next couple years. Their mettle will be tested.
— Scott Clavenna, Director of Research, Light Reading
http://www.lightreading.com
