Scoring Rules & Calibration

The idea that a forecaster who is always 100% certain is the most useful is a myth. Such forecasters are often overconfident and poorly calibrated—meaning their sense of certainty doesn't match reality. Scoring rules are a way to grade probabilistic predictions, rewarding both accuracy and an honest assessment of uncertainty. Calibration measures whether your '80% confident' predictions actually happen 80% of the time. For example, if a project manager says she's 90% sure a task will be done on Friday, but similar tasks are only completed on time 60% of the time, she's poorly calibrated.

Reflect on this: When I say I'm 'pretty sure' about something, what probability would I actually assign to it?

Scoring rules provide a formal process for evaluating probabilistic judgments. The process begins with an input: a specific, numbered forecast (e.g., "There is a 70% chance of rain tomorrow"). After the event occurs (it either rains or it doesn't), a scoring rule is used to assign a score based on both the forecast and the outcome. Repeating this process over time allows you to assess your overall accuracy and calibration.

A key concept is the proper scoring rule, which is designed to give you the best possible expected score only when you state your true belief. These rules discourage hedging or exaggerating your confidence. The most common is the Brier score, which penalizes you based on how far your probability was from the actual outcome (0 for didn't happen, 1 for did happen).

Calibration is the result of this long-term tracking. A well-calibrated person knows what their internal feeling of "70% certain" actually means in the real world. If you look back at all the times you were 70% confident, you should find you were right about 70% of the time. If you were only right 50% of the time, you are overconfident.

In medicine, a well-calibrated doctor can tell a patient, "There's a 20% chance this is a serious condition." This provides crucial information for deciding on risky tests, which is far more useful than a vague "it's unlikely."

Similarly, a startup founder who says, "We have a 60% chance of hitting our revenue target this quarter," allows investors to manage their expectations and risks appropriately. This is more valuable than blind optimism, as it reflects an honest and measured understanding of the business's uncertainty.

Scoring rules provide a formal mechanism to evaluate and improve probabilistic judgment, moving it from a vague art to a measurable skill. Research pioneered by forecasters and psychologists demonstrates that systematic tracking and feedback with proper scoring rules, like the Brier score, significantly improves a person's calibration and accuracy over time.

• Tetlock & Gardner (2015) used Brier scores to identify "Superforecasters," finding that frequent feedback with these scores was a key practice that enabled them to consistently outperform intelligence analysts with access to classified information. [1]
• Gneiting & Raftery (2007) provided a foundational review of scoring rules, distinguishing between calibration (the correspondence between forecasts and reality) and sharpness (the concentration of the predictive distributions), arguing that a good forecast must have both. [2]
• Poses et al. (1985) found that even experienced physicians were often poorly calibrated, showing systematic overconfidence in their diagnostic probability estimates, but that this could be improved with formal feedback and training. [3]

Scoring rules and calibration are powerful but not universally applicable. Their primary limitation is that they require clear, specific, and verifiable outcomes. They are difficult to apply to vague predictions ("the economy will improve") or forecasts so far in the future that feedback is impractical.

Furthermore, good calibration alone is insufficient. One could be perfectly calibrated by always assigning the base rate (e.g., a 50% chance for a coin flip), but this forecast is not very informative or 'sharp'. The goal is to be both well-calibrated and make confident (sharp) predictions when justified by evidence.