Brier Score

The Brier score is a system that grades probability predictions. It works like a penalty score for forecasters, where lower is better, rewarding them for being both correct and confident. A forecaster gets a small penalty for confidently predicting something that happens, but a huge penalty for confidently predicting something that doesn't happen. For example, a weather app that predicts a 90% chance of rain when it rains is scored better than one that only predicts a 60% chance. Over time, the average of these scores reveals who is truly the most accurate and well-calibrated forecaster.

Try this: How well-calibrated are the probability estimates I hear in the news or at work?

The Brier score provides a precise way to evaluate probabilistic forecasts by measuring the mean squared error between predictions and actual outcomes. The process rewards accuracy and calibration—how well a forecaster's probabilities match reality over time.

The calculation for a single forecast is straightforward. First, an outcome is coded as 1 if it happens and 0 if it does not. The score is then calculated by taking the difference between the forecast probability and the actual outcome, and then squaring that difference. A perfect score is 0 (e.g., predicting 100% for an event that happens), and the worst possible score is 1 (e.g., predicting 100% for an event that does not happen).

Let's apply this. If a project manager predicts an 80% chance of success and the project succeeds (coded as 1), the difference between the forecast (0.8) and outcome (1) is -0.2. Squaring this difference results in a Brier score of 0.04, a small penalty. If the project had failed (coded as 0), the difference would be 0.8. Squaring this gives a score of 0.64, a much larger penalty for being confidently wrong.

To evaluate a forecaster, you don't look at one prediction in isolation. Instead, you average their Brier scores across many different forecasts. A forecaster with a consistently low average Brier score is more reliable than one with a high score. This is used everywhere from evaluating medical diagnostic models to assessing geopolitical risk analysts.

For example, a hospital could use the Brier score to compare two different AI systems that predict the likelihood of disease. By tracking their scores across thousands of patients, the hospital can determine which system provides more reliable probabilities to its doctors, leading to better treatment decisions.

The Brier score for a single binary event is formally defined as (p − o)², where 'p' is the forecast probability and 'o' is the outcome (1 if the event occurred, 0 otherwise). It is a type of 'proper scoring rule,' a class of metrics designed to incentivize honest and accurate reporting of probabilities [1]. Proper scoring rules are foundational to modern forecasting evaluation for several reasons:

• They ensure a forecaster achieves their best possible expected score only by stating their genuine belief about an event's likelihood, preventing strategic guessing or hedging [3]. (2007)
• They provide a standardized method for comparing the performance of different models or human forecasters on the same set of problems, as seen in fields from meteorology to expert forecasting tournaments [2]. (2015)

The Brier score is not always intuitive; a score of 0.15 has little meaning in isolation and is best used for comparing forecasters or tracking performance over time. A key limitation is its sensitivity to the event's base rate (or 'climatology'). For very rare or common events, a forecaster can achieve a seemingly good (low) score simply by always predicting the base rate (e.g., 1% chance), showing little actual forecasting skill [4]. Because of this, the Brier score is most informative when used to compare forecasters on the same set of events, as their scores will share the same base-rate component [5]. More advanced analysis often decomposes the score into components of reliability, resolution, and uncertainty to better isolate a forecaster's skill [4].