Are Pharmaceutical Probability of Success Benchmarks Useful?

While teaching the Portfolio Management Course at Stanford (here), Jay Anderson of Eli Lilly offered a provokative tidbit during his guest lecture. Jay had conducted a 13-year study of the assessed probability of success for all Lilly’s drugs in development and compared then to the actual frequency of success.

His assessment committee is very well calibrated, for example, when they say “50% chance of success” the actual success occurs about half the time. Jay compared this result to industry benchmarks. He found that they had almost no predictive power. In other words, there is no relationship between what the benchmark says and the actual frequency of Lilly’s successes. Or as Jay politely puts it “The null hypothesis that the benchmarks have no predictive power cannot be rejected.”

Yet most companies use benchmarks as the basis of their probabilities! Jay points out that this practice misses the considerable private information that firms have about their compounds.

So what is the right role of Benchmarks in assessing probability of success? Food for thought.