Segmenting Test Results

Segmenting test results means breaking your data into meaningful subgroups instead of looking at overall averages. Imagine a medication that helps 70% of patients but harms the remaining 30%—the average effect would look like it does nothing, when in reality it has powerful but opposite effects on different groups. Similarly, that checkout change might have helped mobile users while hurting desktop users, canceling out in the overall average. By looking at subgroups like device type, geography, or customer tenure, you can find effects that averages hide entirely. The correct answer to the hook is that it depends on subgroups—the team should have segmented the data to see if different customer groups responded differently before rolling back. Check this: Next time you see flat results, ask whether you've segmented by meaningful characteristics.

Segmenting works by dividing your experiment data into subgroups based on characteristics that might matter: device type, location, new vs. returning users, traffic source, or past behavior. The process involves first running your experiment to get overall results, then forming hypotheses about which groups might respond differently, and finally analyzing each subgroup separately. When effects differ across groups, the overall average can mask both positive and negative impacts.

The danger of Simpson's Paradox: This statistical phenomenon occurs when a trend appears in several different groups of data but disappears or reverses when these groups are combined. The checkout flow change might have increased mobile conversion by 15% while decreasing desktop conversion by 10%—if mobile users are 40% of your traffic and desktop users are 60%, the overall effect could appear neutral or even negative despite helping a large and growing user segment.

Practical segmentation framework: Start with dimensions you already track. For e-commerce, segment by device type, first-time vs. returning customers, and purchase category. For B2B software, segment by company size, role (decision-maker vs. user), and geographic region. The key is to plan your segments before running the experiment to avoid cherry-picking results afterward. Document your segmentation hypotheses upfront, then test them systematically.

When segmentation matters most: Segmentation is most valuable when you have heterogeneous users with different needs or contexts. A pricing page change might attract price-sensitive new customers while alienating loyal power users. A feature redesign might help beginners but frustrate experts. In these cases, segmentation helps you decide whether to launch different experiences for different segments rather than a one-size-fits-all solution.

Research on subgroup analysis in experiments reveals both its power and its pitfalls. Kohavi et al. (2020) emphasize that proper segmentation requires pre-specifying subgroups to avoid false discoveries: data scientists can always find some segment that shows positive results by chance if they slice the data enough ways after the fact [1]. Gelman (2017) demonstrates how multilevel modeling can improve subgroup estimates by borrowing strength across segments, reducing the high variance that plagues small subgroup analyses [2].

• Kohavi et al. (2020): Trustworthy online controlled experiments pre-specify segments and adjust for multiple comparisons to avoid finding spurious effects through data dredging [1].
• Gelman & Hill (2007): Multilevel (hierarchical) models provide partial pooling between segment and overall estimates, balancing segment-specific learning with general patterns [3].
• Hsu et al. (2020): Subgroup analysis requires larger sample sizes than overall experiments to detect true interaction effects, as power is limited by the smallest segment of interest [4].

Segmenting introduces statistical challenges: smaller sample sizes in each subgroup increase variance and reduce power, making real effects harder to detect. The more segments you create, the more likely you are to find false positives by chance alone—the multiple comparisons problem. Segmentation hypotheses must be pre-registered before viewing results to avoid "p-hacking" or cherry-picking. Additionally, segments can interact with each other in complex ways; a change might help mobile users from California but hurt mobile users from New York, leading to an explosion of potential segment combinations. Finally, segments may shift over time; what worked last quarter for new users might not work this quarter as your traffic source mix changes.