Hypothesis & Experimentation

14 free guides in this category.

A/B Tests & Guardrails

A validation technique where identical variants are tested to confirm your measurement system works before trusting real experiments.

Learn how to select a single, crucial metric to determine the success of a project or experiment, ensuring clear and focused decision-making.

A tool for understanding the range of plausible values for a measurement, accounting for the uncertainty of using samples.

Learn how guardrail metrics act as safety checks in experiments, ensuring that improvements in one area don't cause unintended harm elsewhere.

A testable prediction that explains the relationship between variables and guides scientific inquiry through structured investigation.

The process of tracking key metrics to ensure a new change doesn't cause unintended harm to the user experience or business goals.

The more tests you run, the more likely you'll find something significant by chance alone—even when nothing real is happening.

A method for simultaneously testing multiple variables to identify which combination produces the best outcome.

Extreme outcomes tend to be followed by more moderate ones, purely by chance.

Breaking down experiment results into meaningful subgroups reveals hidden effects that averages alone can miss.

A statistical measure to help determine if an observed result is a genuine effect or just due to random chance.

Discover why repeatedly checking an experiment's results before it's over can lead you to false conclusions.

Two ways decisions can be wrong: false alarms (seeing something that isn't there) and missed signals (missing what's actually there).

A method comparing two versions to see which performs better, using evidence rather than assumptions.