Forecast Calibration

Forecast calibration measures whether your stated confidence matches reality over time. If you say something is 70% likely, it should happen about 70% of the time when you make that prediction. Think of it like a shower dial: if you set it to "warm," the water should actually feel warm, not freezing or scalding. When predictions are well-calibrated, you can trust the numbers to guide decisions—from carrying an umbrella to investing resources. Check this: Track how often your 80% confident predictions actually come true.

Calibration separates overconfident forecasters from underconfident ones. A well-calibrated predictor who says something is 30% likely will see that event occur approximately 30% of the time across many similar predictions. It's not about being right or wrong on any single forecast—it's about the long-run alignment between probabilities and outcomes.

This matters because uncalibrated forecasts mislead decision-makers. If a doctor consistently says "there's a 10% chance of complications" but complications happen 40% of the time, patients can't make informed choices. Similarly, if a manager estimates "90% chance of on-time delivery" but only hits 60%, stakeholders learn to ignore the numbers entirely.

The good news: calibration can be measured and improved. By recording your confidence level (10%, 50%, 90%) for each prediction and tracking outcomes, you develop a personal calibration curve. Overconfident people see their 70% predictions materializing only 40% of the time; underconfident people see them happen 90% of the time. Awareness is the first step toward alignment.

Calibration research reveals a striking pattern: expertise sometimes correlates with calibration challenges; research shows mixed results depending on domain. This calibration-overconfidence paradox appears across domains—intelligence analysts routinely overestimate the certainty of their judgments compared to their actual track records [1]. Training can help: techniques like outcome feedback and considering reasons why predictions might be wrong reduce overconfidence and improve the alignment between stated probabilities and observed frequencies [1].

Key findings:

• Tetlock (2015): Superforecasters—ordinary people who consistently outperformed intelligence analysts—achieved superior calibration through specific habits, including updating predictions as new information arrived and explicitly tracking numerical accuracy over time [1].

• Lichtenstein & Fischhoff (1977): Demonstrated that people are systematically miscalibrated across knowledge domains, with overconfidence increasing with task difficulty; their experimental protocols established calibration curves as the standard measurement method [2].

Calibration alone doesn't capture everything. A forecaster who always says "50% chance" is perfectly calibrated but useless—every prediction is the same non-committal hedge. Resolution, or the ability to distinguish likely from unlikely events, matters too. The best forecasters have both calibration and resolution. Additionally, calibration assumes stable reference classes: if the world changes dramatically (like during a pandemic), past calibration may not transfer forward without recalibration. Finally, rare events pose measurement challenges—it takes hundreds of "1% risk" predictions to verify whether 1% is actually the right number.