Choosing Primary Metrics

A primary metric is the single most important number that tells you if a change was successful. It's your project's 'north star,' preventing you from getting distracted by data that doesn't matter for your specific goal.

For the new 'Stories' feature, focusing on daily story creators is the best choice because it directly measures how many people are adopting the feature's main purpose: content creation. Choosing one clear goal helps the whole team focus on what truly drives success.

Ask this: 'What one behavior, if it changed, would tell me we've succeeded?'

Choosing a primary metric is the first critical step in measuring the success of any change, from a website redesign to a new personal habit. The goal is to distill a complex outcome into one, sensitive, and understandable number, often called an Overall Evaluation Criterion (OEC).

Key Principles for a Good Primary Metric

A strong primary metric should be:

• Actionable: It should directly relate to a behavior you can influence.
• Sensitive: It must be able to detect the impact of your changes, even if small.
• Understandable: The entire team should be able to grasp what it means and why it matters.

For example, an e-commerce site testing a new checkout button should use conversion rate (purchases per visitor) as its primary metric, not just clicks on the new button. The conversion rate connects the change directly to the ultimate business goal: selling more products.

In another domain, a city government wants to improve traffic flow. A poor primary metric would be 'number of new traffic lights installed.' A much better one is 'average commute time through the city center during peak hours,' as it directly measures the resident's experience.

By committing to one primary metric, you force clarity and alignment. It prevents teams from 'cherry-picking' data that supports a desired outcome and focuses everyone on a shared definition of success. This is then balanced by guardrail metrics to watch for unintended side effects.

The concept of a primary metric, or an Overall Evaluation Criterion (OEC), is a cornerstone of trustworthy online experimentation. Selecting a good OEC is arguably the most critical step, as it defines success and can have massive long-term impact on a product or business. Researchers emphasize that a good metric must not only be statistically sensitive but also causally linked to long-term strategic goals, a connection that is often difficult to establish.

• Kohavi, Tang, & Xu (2020) argue that the OEC must be a quantitative measure of an experiment's objective and should align with long-term value, cautioning against optimizing for short-term metrics that might be detrimental over time. [1]
• Deng & Shi (2016) from Microsoft detail a data-driven process for developing metrics, highlighting that good metrics should be sensitive, trustworthy, and interpretable, often requiring a combination of domain knowledge and statistical analysis. [2]
• Casey Winters of Reforge explains that a North Star Metric should represent the core value delivered to customers, arguing that a focus on this single metric aligns the entire company towards sustainable, product-led growth. [3] (n.d.)

Focusing on a single primary metric can create tunnel vision. Teams might inadvertently harm other aspects of the user experience while chasing their main target, a problem known as 'metric fixation.' For this reason, primary metrics must be paired with guardrail metrics (e.g., latency, unsubscribes, support contacts) to monitor for negative consequences.

Furthermore, a primary metric that is effective in the short term may not correlate with long-term success. For instance, increasing email notifications might boost daily engagement (a primary metric) but lead to user fatigue and uninstalls over months.