Prior Elicitation Techniques

Prior elicitation is a careful way to turn what people already know into a clear, testable starting point for decisions. The “priors = bias” claim is a myth when you use structured methods that surface assumptions and check them. Example: a hospital team estimates the chance a new device fails in year one by asking experts for low, best, and high values, then documenting why.

Try this: For a forecast you care about, write lower, best guess, and upper values you’d be 90% sure contain the truth.

How it works (Input → Process → Output):

• Input: define the exact quantity (what, where, timeframe), pick suitable experts, and gather any baseline facts.

• Process: train briefly on common biases, pick a questioning format (e.g., percent-chance, quantiles, or frequency), and use a stepwise protocol: anchor with definitions, elicit ranges, probe reasoning, give feedback, and iterate.

• Aggregate: turn each expert’s judgments into a coherent belief curve, check calibration or realism, and combine experts (equal or performance-weighted). Document everything.

• Output: a transparent prior plus a sensitivity check (how results change under alternative reasonable priors).

Examples: a product manager estimates first-month churn by eliciting a 5–95% range from support, sales, and data leads; a city planner quantifies flood risk from engineers using a structured workshop; an HR team forecasts time-to-fill for critical roles via a short, remote elicitation.

Strongest when data are scarce or shifting, experts are diverse, and the process includes training, feedback, and iteration. Variants include individual vs group sessions, behavioral aggregation (discussion to consensus) vs mathematical pooling (averaging), and performance-weighting when calibration data exist.

Research shows that structured elicitation improves transparency and can outperform ad hoc judgments, especially with scarce data. Protocol choices (format, feedback, aggregation) strongly affect quality.

• Garthwaite, Kadane, and O’Hagan (2005): Reviews elicitation formats and fitting methods; training and feedback reduce overconfidence and improve reliability [2].
• O’Hagan et al. (2006): Stepwise protocols and clear definitions mitigate biases; decomposition and iteration enhance accuracy [1].
• Cooke (1991): Performance-weighted combinations based on calibration and informativeness often beat equal-weight averages [3].
• NICE TSD (2014): Practical guidance (incl. SHELF) shows improved decisions in health technology assessments when elicitations are documented and sensitivity-tested [5].

• Weak when abundant, high-quality data exist—let data dominate.
• Vulnerable to overconfidence, anchoring, groupthink, and poor expert selection.
• Performance-weighting needs calibration questions; without them, equal weighting may be safer.
• Adversarial or politicized contexts can skew inputs; use blinded, independent elicitations.
• Results are assumption-dependent; always run sensitivity analyses.