I read the book SuperForecasting: the art and science of prediction by P. E. Tetlock & D. Gardner. Their Good Judgement Project has been running for years now and the book is the results of their experiments. I thought it was a great book.
But it also got me to thinking, how can industry analysts do a better job at forecasting storage trends and events?
Impossible to judge most analyst forecasts
One thing the book mentioned was that typically analyst/pundit forecasts are too infrequent, vague and time independent to be judge-able as to their accuracy. I have committed this fault as much as anyone in this blog and on our GreyBeards on Storage podcast (e.g. see our Yearend podcast videos...).
What do we need to do differently?
The experiments documented in the book show us the way. One suggestion is to start putting time durations/limits on all forecasts so that we can better assess analyst accuracy. The other is to start estimating a probability for a forecast and updating your estimate periodically when new information becomes available. Another is to document your rational for making your forecast. Also, do post mortems on both correct and incorrect forecasts to learn how to forecast better.
Finally, make more frequent forecasts so that accuracy can be assessed statistically. The book discusses Brier scores as a way of scoring the accuracy of forecasters.
How to be better forecasters?
In the back of the book the author's publish a list of helpful hints or guidelines to better forecasting which I will summarize here (read the book for more information):
- Triage - focus on questions where your work will pay off. For example, try not to forecast anything that's beyond say 5 years out, because there's just too much randomness that can impact results.
- Split intractable problems into tractable ones - the author calls this Fermizing (after the physicist) who loved to ballpark answers to hard questions by breaking them down into easier questions to answer. So decompose problems into simpler (answerable) problems.
- Balance inside and outside views - search for comparisons (outside) that can be made to help estimate unique events and balance this against your own knowledge/opinions (inside) on the question.
- Balance over- and under-reacting to new evidence - as forecasts are updated periodically, new evidence should impact your forecasts. But a balance has to be struck as to how much new evidence should change forecasts.
- Search for clashing forces at work - in storage there are many ways to store data and perform faster IO. Search out all the alternatives, especially ones that can critically impact your forecast.
- Distinguish all degrees of uncertainty - there are many degrees of knowability, try to be as nuanced as you can and properly aggregate your uncertainty(ies) across aspects of the question to create a better overall forecast.
- Balance under/over confidence, prudence/decisiveness - rushing to judgement can be as bad as dawdling too long. You must get better at both calibration (how accurate multiple forecasts are) and resolution (decisiveness in forecasts). For calibration think weather rain forecasts, if rain tomorrow is 80% probably then over time rain probability estimates should be on average correct. Resolution is no guts no glory, if all your estimates are between 0.4 and 0.6 probable, your probably being to conservative to really be effective.
- During post mortems, beware of hindsight bias - e.g., of course we were going to have flash in storage because the price was coming down, controllers were becoming more sophisticated, reliability became good enough, etc., represents hindsight bias. What was known before SSDs came to enterprise storage was much less than this.
There are a few more hints than the above. In the Good Judgement Project, forecasters were put in teams and there's one guideline that deals with how to be better forecasters on teams. Then, there's another that says don't treat these guidelines as gospel. And a third, on trying to balance between over and under compensating for recent errors (which sounds like #4 above).
Again, I would suggest reading the book if you want to learn more.
Storage analysts forecast contest
I think we all want to be better forecasters. At least I think so. So I propose a multi-year long contest, where someone provides a storage question of the week and analyst,s such as myself, provide forecasts. Over time we can score the forecasts by creating a Brier score for each analysts set of forecasts.
I suggest we run the contest for 1 year to see if there's any improvements in forecasting and decide again next year to see if we want to continue.
Question(s) of the week
But the first step in better forecasting is to have more frequent and better questions to forecast against.
I suggest that the analysts community come up with a question of the week. Then, everyone would get one week from publication to record their forecast. Over time as the forecasts come out we can then score analysts in their forecasting ability.
I would propose we use some sort of hash tag to track new questions, "#storage-QoW" might suffice and would stand for Question of the week for storage.
Not sure if one question a week is sufficient but that seems reasonable.
(#Storage-QoW 2015-001): Will 3D XPoint be GA'd in enterprise storage systems within 12 months?
3D XPoint NVM was announced last July by Intel-Micron (wrote a post about here). By enterprise storage I mean enterprise and mid-range class, shared storage systems, that are accessed as block storage via Ethernet or Fibre Channel as SCSI device protocols or as file storage using SMB or NFS file access protocols. By 12 months I mean by EoD 12/8/2016. By GA'd, I mean announced as generally available and sellable in any of the major IT regions of the world (USA, Europe, Asia, or Middle East).
I hope to have my prediction in by next Monday with the next QoW as well.
Anyone interested in participating please email me at Ray [at] SilvertonConsulting <dot> com and put QoW somewhere in the title. I will keep actual names anonymous unless told otherwise. Brier scores will be calculated starting after the 12th forecast.
Please email me your forecasts. Initial forecasts need to be in by one week after the QoW goes live. You can update your forecasts at any time.
Forecasts should be of the form "[YES|NO] Probability [0.00 to 0.99]".
Better forecasting demands some documentation of your rational for your forecasts. You don't have to send me your rational but I suggest you document it someplace you can use to refer back to during post mortems.
Let me know if you have any questions and I will try to answer them here
I could use more storage questions...
Comments?
Photo Credits: Renato Guerreiro, Crystalballer
