The status line that started it
“Estimate benchmarked against three comparable projects; all delivered within tolerance.”
That sentence appears in more charters than it should. It has the grammar of rigor. It cites a method. It names a number of comparables. And it is almost always assembled from the only projects anyone can remember without flinching: the ones that finished, the ones that were celebrated, the ones whose names still appear in portfolio decks.
Call it the reference class of the survivors. The forecast looks like an outside view. It is an inside view wearing the outside view’s coat.
This is not a story about bad arithmetic. The arithmetic is usually fine. The bias enters earlier, at the step where someone decides which past projects count as “comparable.” That decision is rarely written down, rarely challenged, and almost never audited. It is the softest part of the forecast and the part that carries the most weight.
What the outside view is supposed to do
Reference class forecasting is a method for positioning a proposed project inside the statistical distribution of outcomes from comparable projects that have already concluded. As Park describes it, the method “positions a proposed action in the statistical distribution of the outcomes of comparable but already concluded actions” (Park, EJTIR 21(2), 2021, p. 122).
HM Treasury’s supplementary guidance on optimism bias makes the same move in procedural language: explicit adjustments to cost, benefit, and duration estimates “should be based on data from past or similar projects, and adjusted for the unique characteristics of the project in hand” (HM Treasury, Green Book supplementary guidance: optimism bias, 2013). The guidance also states the premise plainly: “Project appraisers have the tendency to be over optimistic.”
The logic is sound. If your estimate is built from the specifics of your project, you will over-weight what you know and under-weight what you don’t. The outside view forces you to look at what actually happened to projects like yours. It is a discipline against the inside view’s natural optimism.
But notice what the method depends on: a class of comparable, concluded projects. The method does not tell you how to build that class. It assumes the class is already honest. That assumption is where the failure lives.
The tell: how the class gets curated
In practice, analog lists are assembled from memory, from portfolio decks, from the projects people are proud to name in a steering committee. The result is a class that is large enough to look rigorous and narrow enough to exclude the failures.
Three comparables. All completed. All within tolerance. The appendix looks like evidence. It is a guest list.
Park notes that there is “no standard practice code for reference class forecasting” and that implementation varies (Park, 2021, p. 123). That variation is not a minor methodological wrinkle. It means the class definition is left to whoever is assembling the forecast, and that person is usually the same person who wants the forecast to be approved.
The outside view, in Park’s summary, “totally ignores the specifics of the current project” and instead looks at “the history of a class of comparable projects” (Park, 2021, p. 122). That is the method’s strength and its vulnerability. If the class is wrong, the method is wrong, and it will be wrong with the confidence of a statistical procedure.
Composite artifact: the analog appendix
The following is a composite, assembled from the kinds of charter appendices that appear in post-mortem files. It is not a real project record.
Appendix C — Analog Projects (Reference Class)
Analog 1: Customer portal replatform, 2019. Completed. Delivered 2 weeks late, within contingency. Cited as “comparable scope and team size.”
Analog 2: Billing system migration, 2020. Completed. Delivered on schedule. Cited as “comparable integration complexity.”
Analog 3: Data warehouse consolidation, 2021. Completed. Delivered under budget. Cited as “comparable vendor dependency.”
Note: Cancelled projects excluded as not comparable. Projects with material scope changes excluded as not comparable. Projects with sponsor turnover excluded as not comparable.
Read the footnote again. The exclusions are not random. They remove exactly the projects whose outcomes would widen the distribution: the cancelled, the re-scoped, the ones whose sponsors left and whose estimates were never tested against reality. What remains is a class of survivors. The forecast is then positioned against that class, and the class has no tail.
This is not fraud. It is curation. The people who assemble these appendices are usually trying to be helpful. They are selecting projects that “feel” comparable. But the feeling is shaped by what is visible, and what is visible is what finished.
Second composite artifact: the metadata
Schedule files carry metadata. Most teams never look at it. In a forensic review, it is one of the cheapest instruments available.
File: Forecast_v7_final.xlsx
Last modified: 14 March, 16:42
Modified by: Sponsor’s office
Analog tab last edited: 14 March, 16:38
Forecast due: 15 March, 09:00
The analog list was edited four minutes before the forecast was finalized, the day before it was due, by the sponsor’s office. That does not prove anything on its own. It is a provenance signal. It tells you who touched the class and when. In a post-mortem, that signal is worth more than a hundred pages of methodology.
The question is not whether the sponsor’s office acted in bad faith. The question is whether anyone independent reviewed the class definition before the forecast was signed. In most cases, the answer is no.
Why the class definition is the whole game
The arithmetic of reference class forecasting is straightforward: identify the class, establish the distribution, position the project. The first step is the one that determines the answer. The second and third steps are mechanical.
If the class contains only projects that finished within tolerance, the distribution is narrow and the forecast will be optimistic. If the class contains only projects that were approved, the distribution excludes the projects that were never approved because their estimates were honest. If the class contains only projects that the sponsor can name, it excludes the projects that were cancelled, absorbed, or quietly written off.
Park reports that a before-and-after comparison in the UK showed average cost overrun declining from 38% to 5% following the introduction of reference class forecasting (Park, 2021, p. 120). That is a striking number. It is also a before-and-after comparison, not a controlled causal estimate, and it does not tell us whether the classes used in those forecasts were built from survivors or from the full population. The improvement is real in the data Park reports. The mechanism is not fully identified.
This matters because the method’s credibility depends on the class being representative. If the class is curated, the method can produce a confident forecast that is no better than the inside view it was meant to replace. The procedure looks like discipline. The inputs are still a wish.
The class audit: a procedural close
The fix is not a better model. It is a documented class definition and an audit trail that survives the forecast. Five steps, in order.
1. Require the class to include cancelled and overrun projects, or state why not
The default should be inclusion. If a cancelled project is excluded, the exclusion rule must be written down and justified. “Not comparable” is not a justification. It is a label. The audit should ask: what specifically made it not comparable, and would that same characteristic apply to the current project?
2. Name who assembled the class and when
Provenance is not bureaucracy. It is the difference between a forecast and an assertion. Record the name, the date, and the version of the analog list. If the list was edited after the forecast was drafted, that edit should be visible in the record.
3. Record the exclusion rule in writing
Every class has an exclusion rule, even if it is implicit. Make it explicit. “Projects with scope changes greater than 20% were excluded” is a rule. “Projects that didn’t feel comparable” is not. The rule should be stated before the class is assembled, not after the forecast is known.
4. Compare the spread, not just the mean
A class of three successes has no spread worth positioning against. The narrowness is itself a warning sign. If the class’s outcomes cluster tightly around “on time, on budget,” ask why. Real project populations have tails. A class without a tail is a class that has been trimmed.
5. Attach the audit to the forecast as an appendix
The audit should travel with the forecast. When the project is reviewed, the class definition should be reviewable. When the project is post-mortemed, the class should be part of the file. The forecast is only as honest as the graveyard it refuses to look at.
What this does not fix
This procedure does not eliminate optimism bias. It does not prevent strategic misrepresentation. It does not guarantee that the class is representative. It does one thing: it makes the class definition visible and contestable before the forecast is signed.
That is a modest improvement. It is also the difference between a forecast that can be audited and one that cannot. The reference class of the survivors is not a statistical error. It is a documentation failure. The fix is documentation.
FAQ
Is this the same as survivorship bias?
It is a specific application of it. Survivorship bias is the general error of drawing conclusions from a set that has been filtered by survival. The reference class of the survivors is the version that appears in project forecasting: the analog class is filtered by completion, approval, or visibility, and the forecast is then positioned against the filtered set.
Does reference class forecasting always fail this way?
No. The method can work when the class is built from a representative population and the exclusion rules are documented. Park’s account of the UK experience suggests the method can be associated with improved outcomes, though the causal mechanism is not fully established. The failure mode described here is a class-definition failure, not a method failure.
What if we don’t have cancelled projects in our records?
Then say so. The absence of cancelled projects in the record is itself a finding. It may mean the organization has not retained them, or it may mean they were never formally cancelled. Either way, the forecast should state that the class excludes cancelled projects because none were available, not because they were deemed not comparable.
Who should assemble the class?
Someone who is not the forecast’s sponsor. The class definition should be reviewable by an independent party before the forecast is finalized. If that is not possible, the provenance record should at least make the sponsor’s involvement visible.
How does this relate to the Green Book’s optimism bias guidance?
HM Treasury’s guidance instructs appraisers to base adjustments on “data from past or similar projects” and to adjust for the unique characteristics of the project in hand (HM Treasury, 2013). The guidance assumes the past-project data is representative. The class audit is a procedure for testing that assumption before the adjustment is applied.
Sources
- HM Treasury, Green Book supplementary guidance: optimism bias, 21 April 2013. https://www.gov.uk/government/publications/green-book-supplementary-guidance-optimism-bias
- Park, J. E., “Curbing cost overruns in infrastructure investment: Has reference class forecasting delivered its promised success?” European Journal of Transport and Infrastructure Research, 21(2), 2021, pp. 120–136. https://journals.open.tudelft.nl/ejtir/article/download/5504/5000/18137