The Peeking Problem

The 'Peeking Problem' is the mistake of repeatedly checking an experiment's results and stopping early as soon as they look promising. This dramatically increases your chance of being fooled by random luck, making you think a temporary fluke is a real improvement. For example, if you check a new app feature's performance every hour, you might catch a brief, random spike in engagement and wrongly conclude the feature is a big success.

Try this: Before starting any test, decide on the duration or sample size and stick to it.

The peeking problem occurs because each time you check your results for statistical significance, you're conducting a new, unplanned statistical test. Each test has a small chance of producing a false positive—showing an effect that isn't really there. When you peek multiple times, these small chances add up, making it much more likely you'll eventually see a 'significant' result just by random chance.

Imagine flipping a coin. You wouldn't be surprised to see a brief streak of three heads in a row if you flipped it 100 times. Stopping your experiment on that lucky streak is exactly what peeking does—it mistakes normal random variation for a meaningful pattern.

In business, a marketing team might peek at daily ad performance. They might see that Ad A is 'winning' on Tuesday and shift the whole budget to it, only to find that Ad B would have performed better over the full week. The initial result was just noise.

Similarly, in clinical trials, peeking could lead researchers to conclude a new drug is effective when it isn't, based on an early cluster of positive outcomes. To prevent this, trials use strict protocols that either forbid interim analysis or use special statistical methods that account for it, because the ethical stakes are so high.

The peeking problem, also known as 'optional stopping,' inflates the Type I error rate (false positives) by allowing random noise to be misinterpreted as a real effect. Each peek acts as an uncorrected statistical test, causing the chances of a false positive to accumulate.

• Continuously monitoring results can dramatically inflate a nominal 5% false positive rate to over 15% after just ten peeks, a form of p-hacking. [1] [3] (2023)
• To allow for early stopping without this error, 'group sequential methods' pre-plan a small number of interim analyses with adjusted significance thresholds to maintain the overall error rate. [2] (2000)
• For most practitioners, the simplest solution is to pre-commit to a fixed sample size or test duration, or to use a testing platform with built-in sequential testing methods that correct for this. [4] (n.d.)

The peeking problem is primarily a concern within frequentist statistics, the framework for most standard A/B tests. In Bayesian statistics, the issue is viewed differently. Bayesian methods can be more robust to optional stopping because they update the probability of a hypothesis as new data arrives. However, this is not a 'free pass' to peek indiscriminately; it requires a well-defined model, appropriate priors, and pre-specified decision rules, such as stopping only when the evidence (e.g., a Bayes Factor) reaches a certain threshold. [5] In rare frequentist cases where an early result is overwhelmingly positive or negative, stopping may be a practical or ethical necessity, but this requires a much higher bar for evidence than a conventional significance threshold.