Calibration Training

Calibration training is a process for aligning your subjective confidence with your objective accuracy. The goal is simple: when you say you're 80% sure about something, you should be right about 80% of the time. For example, a well-calibrated weather forecaster who predicts an 80% chance of rain will see it rain on 8 out of 10 of those days. By the way, Rome is farther north than New York City—how well did your confidence level match that fact?

Check this: For your next small prediction, write down your confidence level as a percentage and see how it turns out.

Calibration training works by creating a structured feedback loop between your judgments and reality. It's a systematic process to turn vague feelings of certainty into a reliable tool for forecasting.

The process involves three core steps:

1. Forecast: Make a specific prediction and assign a numerical probability to it. Instead of saying something "will probably happen," say there's a "75% chance."
2. Record & Observe: Write down your forecast and the reasoning behind it. Then, wait for the actual outcome to occur.
3. Score & Adjust: Compare the outcome to your forecast. If you were 90% sure but wrong, you were overconfident. If you were 60% sure and right, you might be under-confident. Over time, you use this feedback to adjust your internal sense of confidence.

A project manager can use this to estimate deadlines. By tracking her confidence (e.g., "80% sure we'll ship on Tuesday") against actual ship dates, she can learn if she's consistently over-optimistic and adjust her planning and communication accordingly.

Similarly, a doctor diagnosing an illness can assign probabilities to different conditions. By reviewing patient outcomes, they can refine their diagnostic intuition, reducing the risk of overconfidence in a common diagnosis while overlooking a rarer one.

Research shows that most people are poorly calibrated—typically overconfident—but that this cognitive bias can be corrected through deliberate practice. The process involves making numerous probabilistic forecasts and receiving clear, timely feedback. This method is a cornerstone of developing true forecasting expertise, transforming it from an innate talent into a learnable skill.

• Tetlock & Gardner (2015) found that elite "superforecasters" are not defined by who they are, but by what they do: they are actively open-minded, intellectually humble, and constantly work to improve their calibration through practice and feedback [1].
• Fischhoff, Slovic, & Lichtenstein (1977) conducted foundational research demonstrating that overconfidence is a robust and pervasive bias, but also that intensive feedback on the accuracy of probability judgments could dramatically improve calibration and reduce overconfidence [2].
• Moore & Healy (2008) distinguished between different forms of overconfidence, including overestimation (believing you are better than you are) and overprecision (being too certain your beliefs are correct), both of which can be addressed by targeted calibration exercises [3].

Calibration is highly domain-specific; being a well-calibrated weather forecaster does not make you a well-calibrated political pundit. The training requires frequent, unambiguous, and timely feedback, which is often unavailable in complex, "wicked" domains with long feedback loops (e.g., strategic business decisions or foreign policy). Furthermore, without a disciplined commitment to recording and reviewing forecasts, the training is ineffective.