The 1-Minute Lesson That Fixes 90% of A/B Testing Confusion
Why your "10% improvement" might actually be 0.1 percentage points
Here's the moment everything clicks: You run an A/B test, your tool shows "15% improvement," and you celebrate. Then you check the actual numbers and your conversion rate only went from 2% to 2.3%.
What happened? You just learned the difference between relative and absolute improvements the hard way.
The Two Ways to Measure the Same Result
When conversion improves from 2% to 2.2%:
Absolute improvement: 0.2 percentage points
Relative improvement: 10% ((0.2 ÷ 2.0) × 100)
Same result. Completely different story.
Why This Breaks Your Brain
Most A/B testing tools use relative improvements when you set your Minimum Detectable Effect (MDE). So when you enter "10% MDE," you're not testing for a 10 percentage point jump. You're testing for a 10% boost from whatever your baseline is.
The plot twist: Same MDE setting, wildly different results depending on your starting point.
Email signup (20% baseline): 10% MDE = 22% target (2-point gain)
Homepage conversion (1% baseline): 10% MDE = 1.1% target (0.1-point gain)
One test could add 200 conversions per 1,000 visitors. The other adds 1.
Three Examples That Make This Real
Low-converting landing page (1% baseline, 15% MDE)
Target: 1.15%
Gain: 15 extra conversions per 10,000 visitors
Email campaign (25% baseline, 15% MDE)
Target: 28.75%
Gain: 375 extra conversions per 10,000 visitors
Checkout flow (60% baseline, 15% MDE)
Target: 69%
Gain: 900 extra conversions per 10,000 attempts
Same percentage improvement. Vastly different business impact.
The Three Mistakes Everyone Makes
Mistake 1: Setting 10% MDE and expecting your 5% conversion rate to hit 15% Reality check: You're testing for 5.5%
Mistake 2: Comparing relative improvements across different baselines The trap: Thinking 20% improvement on your homepage beats 8% improvement on checkout Reality: The checkout improvement probably added 10x more revenue
Mistake 3: Getting disappointed by single-digit improvements on high-converting pages Truth: A 3% relative improvement on a 50% conversion rate is massive
When to Use Which Measurement
Relative improvements for:
Setting up tests in your tool
Comparing similar pages
Absolute improvements for:
Calculating business impact
Talking to your boss
Deciding what to implement
Your New Testing Routine
Before every test:
Note baseline rate
Set relative MDE in tool
Calculate absolute target
Estimate business impact
When reporting results: "Conversion improved from 5% to 5.8% — a 0.8 percentage point increase, representing a 16% relative improvement and 80 additional monthly conversions."
The Bottom Line
For business decisions, think in absolute terms. A tiny relative improvement on a high-converting page often beats a massive relative improvement on a page nobody visits.
Master this distinction, and you'll never misinterpret another A/B test result.
Was this helpful? Reply and let me know what A/B testing concept you'd like me to break down next.