Multiple Comparisons Problem

Each time you test for a relationship, there's a small chance you'll find one that doesn't actually exist—called a false positive. When you run many tests on the same data, these small chances add up. With 20 tests using a 5% significance threshold, you'd expect about one false positive even if no real effects exist. This happens everywhere: A/B testing dozens of website changes simultaneously, scanning thousands of genes for disease links, or mining nutrition surveys for correlations. Notice this: Whenever you see someone report many findings at once, ask whether they adjusted for multiple testing.

How False Positives Accumulate

Think of each statistical test as a lottery ticket with a 5% chance of a false positive when you use standard significance thresholds. One test gives you a 5% risk. But run 20 independent tests, and the probability of at least one false positive jumps to about 64%. Run 100 tests, and it's virtually certain you'll find something 'significant' purely by chance. This happens because the tests compound their error rates—each new comparison creates another opportunity for random noise to look like a real pattern.

Where This Shows Up

A/B testing: An e-commerce site tests 15 different button colors, headlines, and page layouts simultaneously. Even if none truly affect sales, the math says there's a 54% chance at least one will appear 'statistically significant.' This leads teams to waste resources implementing changes that looked promising in testing but don't actually work.

Medical research: Genome-wide association studies scan over a million genetic variants looking for disease links. Without correction, they'd report tens of thousands of false associations. The field uses strict corrections like the Bonferroni method or false discovery rate control to combat this.

Nutrition science: Researchers analyzing food survey data might test hundreds of foods against dozens of health outcomes. The infamous 'green jelly beans cause acne' result from xkcd illustrates this: test 20 colors, and one will show a link by chance. This is why you see contradictory nutrition headlines—coffee causes cancer one week, prevents it the next.

Practical Solutions

Adjust your threshold: The Bonferroni correction divides your significance threshold by the number of tests. With 20 comparisons, instead of using 5%, you'd require 0.25% significance. This is conservative but simple.

Control the false discovery rate: Instead of trying to eliminate all false positives, the Benjamini-Hochberg procedure limits the proportion of false discoveries among your findings. This is less conservative and widely used in exploratory research.

Plan fewer tests: Before looking at data, decide which comparisons matter most. Pre-register your analysis plan so you can't keep fishing until you find something. In A/B testing, run one primary test and a small number of planned secondary tests rather than everything you can think of.

The multiple comparisons problem describes how the probability of at least one Type I error (false positive) increases with the number of hypothesis tests performed. When m independent tests are conducted at significance level alpha, the family-wise error rate equals 1 - (1 - alpha)^m, which approaches 1 as m grows large. This means that with sufficient testing, false discoveries become inevitable without correction.

• Ioannidis (2005): Demonstrated that for most study designs and settings, a research finding is more likely to be false than true, with flexibility in design, definitions, outcomes, and analytical modes contributing to bias that amplifies false positives. [1]

• Benjamini and Hochberg (1995): Introduced the false discovery rate (FDR) as a less conservative alternative to family-wise error rate control, allowing researchers to identify candidate positives for follow-up while controlling the expected proportion of false discoveries. [2]

• Miller (1981): Established foundational theory for simultaneous statistical inference, documenting how multiple comparison procedures developed from the work of Tukey, Scheffe, and others in the 1950s to address the inflation of Type I error rates. [3]

• Noble (2009): Clarified that in large-scale testing (e.g., genomics with millions of comparisons), family-wise error rate control may be too stringent, making false discovery rate control the preferred approach for exploratory studies where findings can be re-tested. [4]

Not all situations require multiple testing correction. If you have strong prior evidence for specific hypotheses before collecting data, you can treat them as confirmatory tests without correction. The problem mainly affects exploratory analyses where you're hunting for patterns without pre-specified expectations.

Some corrections are overly conservative. The Bonferroni method can make it nearly impossible to detect real but small effects, especially with hundreds of tests. This leads to Type II errors (false negatives)—missing real discoveries. False discovery rate methods like Benjamini-Hochberg offer a practical middle ground.

Tests are rarely independent in practice. Most real-world datasets have correlated variables, which the simple formulas don't fully capture. However, corrections that assume independence still tend to be conservative—they may over-correct, but they don't under-correct.