Sequential Testing & SPRT

The idea that you must set a sample size in advance is a myth for certain statistical methods. Sequential testing allows you to analyze results as data comes in and stop the moment you have a confident answer. Instead of committing to survey 1,000 people, you check after each response to see if the evidence is strong enough to favor one conclusion over another. This is like tasting soup as you cook; you add salt, stir, taste, and repeat, stopping as soon as it's perfect, not after a pre-decided number of tastes.

Ask this: Could I make this decision with less information if I checked my progress along the way?

Sequential testing provides a framework for making decisions under uncertainty without committing to a fixed sample size upfront. This is particularly useful when data collection is expensive or time-consuming. The process begins by defining two competing hypotheses (e.g., 'New feature increases engagement' vs. 'New feature has no effect') and the acceptable error rates for the decision.

Based on these inputs, two decision boundaries are established before the test begins. After each new piece of data is collected, a statistical score—typically a 'likelihood ratio' that measures the cumulative evidence—is updated. This running score is then compared against the pre-set boundaries.

If the score crosses the upper boundary, the test stops and the first hypothesis is accepted. If it crosses the lower boundary, the test stops and the second hypothesis is accepted. If the score remains between the boundaries, data collection continues. This methodical process ensures that any decision to stop early is statistically valid and not just a result of premature 'peeking' at the data.

This method yields a conclusion that meets a desired level of confidence, often much faster and with fewer resources than a traditional fixed-sample test. For example, in manufacturing, a factory can test products one by one. If the first five are flawless, the test might signal to stop and approve the batch. Similarly, an e-commerce site can test a new layout and stop the moment one version's outperformance crosses a pre-defined evidence threshold, rather than waiting for a full week of data.

The Sequential Probability Ratio Test (SPRT), developed by Abraham Wald during WWII, is provably the most efficient statistical test for deciding between two hypotheses, requiring the fewest samples on average for a given level of accuracy. The core principle of accumulating evidence until a decision threshold is crossed has proven fundamental in many fields.

• Wald (1945) first formulated the SPRT to test munitions, showing it minimizes the average sample number needed for a decision compared to any fixed-sample test with the same error probabilities. [1]
• Modern tech companies use sequential methods for large-scale A/B testing. Rather than naively peeking, these methods use 'always-valid p-values' or pre-specified stopping boundaries that allow for continuous monitoring and valid early stopping, which accelerates innovation by reducing the time needed to get a reliable answer. [2] (2017)
• Research in neuroscience suggests the brain makes perceptual decisions similarly. Neurons appear to accumulate sensory evidence over time, with a decision triggered once this signal reaches a critical threshold, mirroring the logic of sequential analysis. [3] (2007)

While efficient, sequential testing has limitations. First, the final sample size is not known in advance, which can complicate planning and budgeting for experiments. Second, the classic SPRT is designed for binary choices (A vs. B) and becomes more complex for multiple alternatives. Finally, the test's duration can be highly sensitive to the initial data stream; a run of ambiguous early data can sometimes prolong the test unnecessarily.