4 Costly A/B Testing Mistakes You’re Probably Making
It’s 3 PM on a Thursday. A product manager at a Series B SaaS company opens her A/B testing dashboard for the fourth time that day, a half-drunk cold brew beside her laptop. The screen reads: Variant B, +8.3% conversion lift, 96% statistical significance.
She screenshots the result. Posts it in the #product-wins Slack channel with a party emoji. The head of engineering replies with a thumbs-up and starts planning the rollout sprint.
Here’s what the dashboard didn’t show her: if she had waited three more days (the original planned test duration), that significance would have dropped to 74%. The +8.3% lift would have shrunk to +1.2%. Below the noise floor. Not real.
The Peeking Problem: 26% of Your Winners Aren’t Real
Every time you check your A/B test results before the planned end date, you’re running a new statistical test. Not metaphorically. Literally.
- Frequentist significance tests are designed for a single look at a pre-determined sample size.
- Checking results after 100 visitors, 200, 500, then 1,000 creates four separate tests.
- Evan Miller’s analysis shows peeking inflates false positives from 5% to 26.1%.
One in four “winners” is pure noise. Teams waste engineering hours and misreport revenue impact based on these flawed results.
The Power Vacuum: Small Samples, Inflated Effects
Peeking creates false winners. The second sin makes real winners look bigger than they are.
Statistical power determines your test’s ability to detect real effects. Most teams skip power calculations, leading to the “winner’s curse.”
- Underpowered tests produce inflated effect sizes.
- A +8% lift in a small sample might settle at +2% in reality.
- Fix: Run power analysis before testing. Set a Minimum Detectable Effect (MDE) and calculate required sample size.
The Multiple Comparisons Trap
Tracking five metrics at 5% significance? Your false positive rate jumps to 22.6%. Scale to 20 metrics, and it hits 64.2%.
Fix: Declare one primary metric before testing. Use corrections like Benjamini-Hochberg or Holm-Bonferroni if evaluating multiple metrics.
When “Significant” Doesn’t Mean Significant
A test can be statistically significant but practically meaningless. For example, a +1% lift might not justify engineering costs.
- Statistical significance ≠ business impact.
- Always evaluate effect size alongside p-values.
- Ask: Does this change matter to users or revenue?
Pre-Test Checklist for Reliable Results
- Calculate required sample size and MDE before starting.
- Declare one primary metric and 2-3 secondary metrics.
- Use sequential testing tools to avoid peeking.
- Apply multiple comparisons corrections if analyzing multiple metrics.
- Run post-test power analysis to validate results.
Conclusion: Test Smarter, Not Harder
A/B testing is a powerful tool—but only when used correctly. Avoid these four mistakes to save time, money, and engineering resources. Start with a pre-test checklist and choose the right statistical framework (frequentist, Bayesian, or sequential) for your next experiment.
Ready to run better tests? Share this checklist with your team and start testing with confidence.
FAQs
What are the most common A/B testing mistakes?
The top four are: peeking at results, underpowered tests, multiple comparisons, and confusing statistical significance with practical impact.
How does peeking affect A/B test results?
Peeking increases false positives from 5% to 26.1% by running multiple implicit tests during the experiment.
Why do small samples inflate effect sizes?
Underpowered tests detect only exaggerated effects. A +8% lift in a small sample might settle at +2% in reality.
How to avoid the multiple comparisons trap?
Declare one primary metric and use corrections like Benjamini-Hochberg when analyzing multiple metrics.
What’s the difference between statistical and practical significance?
Statistical significance shows a result is unlikely due to chance. Practical significance evaluates whether the effect matters to your business goals.
