Statistical Significance & P-value

It's a common myth that 'statistical significance' means an effect is large or important. Statistical significance does not measure an effect's size or practical importance; it only indicates how unlikely an observed result is if it were due to random chance alone. Think of it as a smoke alarm: it reliably detects that something is happening (it's not just a sensor glitch), but it can't tell you if it's a burnt piece of toast or a house fire. For example, a website might find a 'significant' result that changing a button's color increases clicks by a tiny 0.01%, an effect too small to matter in practice.

Ask this: 'Is this result statistically significant, and is it also practically meaningful?'

Statistical significance is a tool to filter signal from noise. It starts with a default assumption called the 'null hypothesis' — the idea that there is no real effect or difference. For example, a new drug has no effect, or a new website design doesn't change user sign-ups.

The p-value is the probability of seeing your observed data (or even more extreme data) if that null hypothesis were true. A small p-value (typically under 0.05, or 5%) suggests that your observed result would be very surprising if there were truly no effect. This leads you to reject the 'no effect' idea and conclude the result is 'statistically significant'—it's likely not just a random fluke.

Consider an A/B test for a marketing email. The null hypothesis is that two different subject lines perform identically. If you see a big difference in open rates and get a p-value of 0.03, it means there's only a 3% chance you'd see a difference that large if the subject lines were actually the same. You can be reasonably confident the new subject line had a real effect.

However, this tool has limits. A study on thousands of people might find a new diet pill causes a statistically significant weight loss with a p-value of 0.01. But if the average weight loss is only half a pound, the result is not practically useful, despite its statistical significance. The size of the effect matters just as much as the statistics.

The p-value was developed in the 1920s as a way to assess the strength of evidence against a null hypothesis in a single experiment. However, its widespread use as a simple bright-line test for truth has been heavily criticized by statisticians, who advocate for a more holistic view of evidence that includes effect sizes, confidence intervals, and prior knowledge.

• The American Statistical Association (ASA) published a formal statement clarifying that p-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone [1]. (2016)
• Regina Nuzzo highlighted in Nature that the replication crisis in science is partly fueled by the misunderstanding and misuse of p-values, leading researchers to chase statistical significance over robust, meaningful findings [2]. (2014)
• Historically, the concept originated from R.A. Fisher's work as an informal index of evidence, but it was later merged with the Neyman-Pearson hypothesis testing framework, creating the rigid decision rule (e.g., p < 0.05) that is common today and often used without context [3]. (2001)

The concept of statistical significance is frequently misused. Key limitations include:

• P-hacking: Researchers may consciously or unconsciously run many tests and only report the ones that pass the 0.05 threshold, creating false-positive results.
• Ignoring Effect Size: A result can be statistically significant but practically meaningless if the effect is very small.
• The Cliff Effect: Treating p=0.049 as a success and p=0.051 as a failure is a false dichotomy. The evidence is nearly identical in both cases.
• Misinterpretation: A p-value is not the probability that the null hypothesis is true. It is a statement about the data, assuming the null hypothesis is true.