Comment

Why even I don’t believe my own forecasts

Every forecast I’ve read (or compiled) has been wrong, There, I’ve said it. That statement should not be surprising. Instead, we should step back in amazement when a forecast is retrospectively seen to be anywhere near what actually happened!

Let’s assume you do a REALLY simple forecast in Phase 2 for a product’s Peak Year Sales. And by simple, I mean 10 assumptions or variables (market size, market growth, market share achieved, pricing, compliance, competitor impact, etc) – most forecasts have dozens or hundreds of assumptions. Let’s assume our assumptions are spot-on and our accurate forecast delivers PYS of $1B in 2028.

Now let’s look at the real world where assumptions are not accurate. In fact, we’ll assume that each is 95% accurate and that is WAY more accuracy than any forecaster would realistically claim for assumptions for 12 years in the future.

  • So the upside forecast would be $1B x 1,01 x 1.01$ 10 times = $1.62B.
  • And the downside would be $1b x 0.95 x 0.95 10 times = $0.59B.
  • A swing of $1.03 billion!! On a base forecast of $1,000 million… (And yet, I’ve seen forecasts delivered to senior management to the nearest dollar.)

Imagine going into your boss and saying “the base case forecast is somewhere between $0.59B and $1.62B, but to be honest the variance is far wider than that)”. Clear your desk on the way out…

So why do we give so much importance to forecasts? Well, you have to base future investment on something, and they are the best we’ve got.

But even if you’re 100% accurate, you may not be accepted either due to bias or the short term needs of senior management. I once led a team which spent 3 months developing a PYS forecast which we delivered to the Global Commercial Lead, whose reaction was “Not enough. I need at least 30% more than that to justify my sales force and promotional requests. Go away and redo it.” And he then walked out of the room. To my shame, I didn’t protest and instead went away and deliberately delivered a false forecast so he could get his resourcing and I could keep my job. (By the way, history shows that my base case was far more accurate than his “add on 30%”, but by then we were both several jobs down the line.)

I’m going to go out on a limb here and say that every single forecast made in Phase II is wrong. And the reasoning is SO simple:

  • The “best” attrition rate for Phase 3 products is about 25% (i.e. success rate of 75%) for cardiovascular products, which have a registrational success rate of 95%
  • The “worst” rates for Phase 3 survival and registration are for woman’s health at 45% and 55% respectively
  • On the whole, launched drugs have a “1 in 4” chance of breaking even
  • So, the chances of a drug entering Phase 3 going on to make a profit (to fund all the drugs that didn’t) is:
    • Cardiovascular = 75 x 95 x 25 = 18%
    • Women’s health = 45 x 55 x 25 = 6%

And THAT’s for a drug entering Phase 3, imagine what the odds are for drugs in Phases 1 or 2.

Therefore, if you are asked to support a go/ no-go decision in Phase 2 for progress into 3, your reply MUST be:

“Kill it. Kill it now. Take the one billion dollars, go to Monte Carlo and place it all on black. At least then you have a 48.6% chance of winning.”

(By the way, do NOT go to Las Vegas as American roulette wheels have a “00” slot so there are 39 possible slots in which the ball can land, versus 38 in Europe, so your odds plummet to 46.4%!! See http://www.rouletteonline.net/odds/)

And yet this is the problem with forecasting, it is all too tempting to imagine a bright, sparkly future where molecules don’t fail, and compliance is unfeasibly high. Thomas Jefferson said “I like the dreams of the future better than the history of the past” so do your company a favour the next time you’re viewing a forecast by asking few simple questions. Who commissioned the forecast and what is its aim: to capture investment? To ensure a positive “go decision”? Does the forecast’s commissioner stand to benefit in extra kudos or power by being in charge of a product with a mega forecast? Don’t be afraid to say the forecast looks very positive and enquire about the assumptions that drive such positivity (and if the forecast’s owner won’t discuss events and assumptions then walk away quickly…) And always ask about the variance. A one billion forecast should in practice mean 1 billion plus or minus x% - if the owner won’t discuss upper and lower limits of his or her base forecast then once again run for the hills.

I’ll conclude by saying that I’m not being cynical about forecasting. I actually believe in forecasting. Done properly. With the right level of humility and reality. And presented as roughly right (rather than accurately wrong).