What is the most common mistake product managers make when running A/B tests?

Peeking — checking results daily and stopping the moment the test looks statistically significant. This inflates the real false-positive rate well above 5%, because checking repeatedly is effectively running many small tests and keeping the first lucky one. The fix is to commit to a pre-calculated sample size and duration before looking, or use a sequential-testing method designed for safe early stopping.

How do I decide how long an A/B test should run?

Calculate the required sample size before launch using your baseline conversion rate, current traffic, and the smallest lift that would actually be worth shipping. Divide that by your daily traffic to get the run time, and commit to it — an underpowered test doesn’t fail loudly, it just produces a noisy result that’s tempting to misread as a win.

Should I pick my A/B test’s primary metric before or after seeing the results?

Always before. Pre-register one primary metric that determines win or lose, and treat every other metric you track as directional context only. Selecting the "best" metric after the data comes in turns a controlled experiment into a search for a favorable story, and with enough metrics available, something will move by chance alone.

Is it okay to segment A/B test results by device or user type?

Only if you predicted that segment mattered before the test launched. Slicing results into many post-hoc segments until one turns significant will almost always surface a green result somewhere by chance. Pre-register the one or two segments you have a real hypothesis for, and treat any others as an idea for your next test rather than proof from this one.

A/B Testing 101 for Product Managers: How to Run a Test You Can Actually Trust

Product Leader.

We use cookies for analytics and to remember your preferences. "Necessary" is always on.

💬