Calibration Training

Calibration training is a method for making your subjective confidence match your actual accuracy. The first step is ensuring everyone agrees on what '90% confident' means—namely, that you expect to be right 9 out of 10 times. For example, a meteorologist who is well-calibrated and predicts a '70% chance of rain' will be correct on about 7 out of every 10 days they make that forecast. This process helps you and your team avoid costly overconfidence and make more reliable judgments.

Try this: Ask your colleagues what being '80% sure' about something means to them and see how much the answers vary.

Calibration training is a structured feedback loop designed to improve your ability to assess the probability of your own success. The process systematically exposes the gap between what you think you know and what you actually know, allowing you to adjust.

The process follows four key steps:

1. Define the Scale: The group must first agree on a shared, concrete definition for confidence levels. For example, a '90%' confidence rating means that for every ten such predictions made, exactly nine are expected to be correct. Without this shared language, the exercise is meaningless.
2. Generate Predictions: Individuals answer a series of questions with verifiable outcomes, assigning a confidence level to each answer. A sales team might predict which of 20 leads will close this quarter, while a software team might estimate which features will ship bug-free by the deadline, assigning 70%, 80%, or 90% confidence to each.
3. Provide Feedback: Once the outcomes are known, participants review their results. They check their performance at each confidence level. Did the 80%-confident predictions turn out to be correct about 80% of the time? This reveals patterns of over- or under-confidence.
4. Analyze and Adjust: The group discusses the results. Seeing the hard data—'When we were 90% sure, we were only right 60% of the time'—forces an adjustment in how they approach future estimates. This loop of predicting, checking, and adjusting is what builds better judgment over time.

Calibration training is a direct intervention against the overconfidence effect, one of the most robust findings in cognitive psychology. This bias describes our tendency to have subjective confidence in our judgments that is reliably greater than our objective accuracy. The training provides structured feedback, a necessary component for improving metacognitive skills (thinking about our own thinking) and forecasting accuracy.

• Lichtenstein, Fischhoff, & Phillips (1982) provided a foundational review showing that overconfidence is pervasive across domains, from general knowledge questions to expert predictions, establishing the need for corrective procedures like calibration training. [2]
• Moore & Healy (2008) distinguished between three types of overconfidence, suggesting that calibration helps primarily with 'overprecision' (being too certain in the accuracy of one's beliefs) and 'overestimation' (overestimating one's performance). [3]
• Tetlock & Gardner (2015) found that the best forecasters—so-called 'superforecasters'—are not necessarily experts in one domain but are skilled at probabilistic thinking and are relentlessly focused on calibration through constant feedback and adjustment. [1]
• Kahneman et al. (2021) emphasize that regular calibration exercises, even with simple trivia questions, can create a shared language of probability within a team and significantly reduce noise and bias in strategic decisions. [4]

Calibration training is highly effective but has limitations. Its success depends on having a set of questions with clear, unambiguous, and relatively quick feedback; it's less useful for complex, long-term forecasts where outcomes are murky or delayed. The process can also feel tedious, and if not managed well, can lead to disengagement. Finally, the benefits are a perishable skill; calibration requires ongoing practice to maintain, rather than being a one-time 'fix'.