Attrition & Missing Data

When participants drop out of an experiment or researchers can't collect all the data they planned, the people who remain might be different from those who left. This difference can create a biased picture that doesn't represent the true effect. For example, if a workplace training program loses the least motivated employees before the final evaluation, the results will look better than they really are. Check this: Whenever you see study results, ask what proportion of participants completed the study and whether those who dropped out differed from those who stayed.

First, random assignment initially creates comparable groups at the start of an experiment. Then, as the study progresses, some participants leave, skip measurements, or provide incomplete data. The key question is whether this attrition relates to both the treatment and the outcome. If the people who drop out differ systematically from those who remain—such as when sicker patients leave a medical study while healthier patients stay—the final comparison becomes biased because the groups are no longer comparable.

Understanding the mechanism behind missing data helps diagnose the problem. When data is missing completely at random (like a misplaced survey), the remaining sample still represents the original group. When data is missing based on observed characteristics (like younger participants being harder to track), researchers can sometimes adjust statistically. But when the missingness depends on the unmeasured outcome itself—such as depressed people being less likely to respond to depression surveys—the bias cannot be fixed without strong assumptions.

This pattern appears across many domains. In education research, students who struggle are more likely to transfer schools, potentially making a teaching method appear more effective than it truly is. In clinical trials, patients experiencing side effects may drop out of the treatment group while those doing well remain, exaggerating the benefits. In workplace studies, highly engaged employees participate more fully in surveys, skewing satisfaction measures upward. The validity threat is most severe when dropout rates differ between treatment and control groups, or when the reason for dropping out relates to the outcome being measured.

The most reliable safeguard is tracking all participants regardless of completion (intention-to-treat analysis), comparing the characteristics of dropouts versus completers, and conducting sensitivity analyses to test how different assumptions about missing data would change the conclusions.

Attrition bias threatens validity when the probability of missing outcome data depends on both group assignment and the true outcome value. The Cochrane RoB 2 tool assesses this through three signaling questions: whether outcome data are available for nearly all participants, whether evidence suggests the result is unbiased, and whether missingness depends on the true value [1]. Research on missing data mechanisms distinguishes MCAR (missing completely at random), MAR (missing at random given observed data), and MNAR (missing not at random), where only MCAR andMAR permit valid inference without strong assumptions [2]. Differential attrition between experimental arms often indicates MNAR mechanisms that distort effect estimates [3].

• Higgins (2019): Attrition bias is one of five core domains in the RoB 2 tool; bias occurs when missingness in the outcome depends on both intervention group and the true outcome value, making complete-case analysis unreliable [1].
• Little & Rubin (2020): Missing data are classified as MCAR (probability of missingness unrelated to any data), MAR (missingness depends only on observed variables), or MNAR (missingness depends on unobserved values); modern methods like multiple imputation require MAR for valid inference [2].
• Salthouse (2019): In a longitudinal cognitive study of nearly 5,000 adults, participants over 65 who dropped out had significantly lower baseline cognitive functioning than those who continued, confirming selective attrition that restricts generalizability to higher-functioning individuals [3].

Missing data classifications rely on untestable assumptions about the unobserved values—researchers cannot definitively distinguish MAR from MNAR using only the observed data. Even when attrition rates appear similar across groups, the underlying reasons may differ systematically. Sensitivity analyses help but cannot fully resolve the fundamental problem. Additionally, intention-to-treat analysis preserves randomization benefits but cannot eliminate bias from outcome-dependent missingness. Multiple imputation methods, while sophisticated, still require the MAR assumption and correctly specified models.