Confidence Intervals

A confidence interval is a range of plausible values for an unknown quantity, like an average, based on data from a sample. A 95% confidence level does not mean there's a 95% chance the true value is inside this specific interval. Instead, it means that if we repeated the measurement process many times, we would expect 95% of the intervals created to contain the true value. Try this: When a poll reports a result of 45% with a 3% margin of error, what is the full range of plausible outcomes?

A confidence interval is a primary tool for expressing the uncertainty around an estimate derived from a sample. Since a different sample would produce a slightly different result, the interval provides a range that accounts for this 'sampling error,' giving us a sense of the estimate's precision.

The width of this range is determined by two main factors: the confidence level and the sample size. Choosing a higher confidence level, like 99% instead of 95%, requires a wider, less precise interval. To achieve both high confidence and a narrow, precise range, a larger sample size is necessary.

This concept has broad applications. In business, an A/B test might show a new feature increases conversion by 3% with a 95% CI of [1%, 5%], suggesting a genuine improvement. In public policy, a report might state the unemployment rate is 4.2% with a 90% CI of [4.0%, 4.4%], informing officials about the data's precision.

Similarly, in medicine, a study could find a drug reduces cholesterol by 20 points with a 99% CI of [15, 25]. This gives doctors a highly reliable range for the drug's expected effect across the patient population, helping them manage treatment plans effectively.

The concept of confidence intervals was developed by Jerzy Neyman in the 1930s as a cornerstone of frequentist statistics. This framework treats the population parameter as a fixed, unknown constant and the interval as the random variable. The confidence level refers to the long-run success rate of the procedure used to calculate the interval, not the probability of a single, calculated interval containing the parameter. This is often contrasted with the Bayesian concept of a 'credible interval,' which can be interpreted as a probabilistic range for the true value, given the data and a prior belief.

• Neyman (1937): Established the frequentist definition, stating that the parameter is fixed and the interval is random; the 95% refers to the success rate of the estimation procedure over many hypothetical repetitions. [1]
• Morey et al. (2016): Argue that the common misinterpretation of confidence intervals is so tempting because it aligns with a more intuitive, Bayesian desire to state our belief about the parameter itself. They highlight that confidence intervals do not license beliefs about the parameter from a single experiment. [2]
• Cumming & Finch (2005): Promoted the use of CIs over simple p-values for inference, arguing that CIs provide more useful information about the magnitude and precision of an effect, helping researchers make better judgments by 'inference by eye'. [3]

The biggest limitation of confidence intervals is their counter-intuitive frequentist interpretation, which is frequently misunderstood and misstated as a probabilistic statement about the parameter. Furthermore:

• The choice of confidence level (e.g., 95%) is an arbitrary convention. There is no objective reason 95% is better than 94% or 96%.
• The validity of the interval depends on the assumptions of the statistical model being correct (e.g., random sampling, normality of errors).
• A confidence interval does not tell you anything about the probability of a specific value within the interval being the true value; it only provides a plausible range.