The Complete Guide to Minimum Detectable Effect (MDE): How to Select MDE for Every A/B Test
Here's the most misunderstood concept in A/B testing: Minimum Detectable Effect (MDE). Get this wrong, and you'll either run tests forever or miss improvements that could transform your business.
This guide breaks down MDE in easy-to-understand language and shows you exactly how to set it for every test.
How to Actually Consider Business Impact and Timeline: The Most Important MDE Concept
This is the foundation everything else builds on: MDE selection should start with business value, not statistical preferences.
Most teams approach this backwards. They pick an MDE percentage that “feels right” without understanding what it means for their business timeline and revenue impact.
How MDE Affects Test Inclusiveness, Sample Size, and Statistical Errors
The core relationship: Lower MDE = Higher sample size requirements = Longer test duration
Why this happens: To detect smaller differences, you need more data to be confident the difference is real and not random noise.
Statistical error types:
Type I Error (False Positive): Concluding there's an improvement when there isn't
Type II Error (False Negative): Missing a real improvement because your test wasn't sensitive enough
How MDE connects to these errors:
Lower MDE (5-8%):
Advantage: Less likely to miss small improvements (fewer Type II errors)
Cost: Requires massive sample sizes, very long tests
Risk: More likely to never reach statistical significance
Higher MDE (18-25%):
Advantage: Faster tests, more conclusive results
Cost: Will miss improvements smaller than your MDE (more Type II errors for small effects)
Benefit: Less likely to waste time on inconclusive tests
The business tradeoff: You're choosing between the risk of missing small improvements vs. the risk of slow, inconclusive testing.
Start With Business Value, Not Statistical Preferences
Step 1: Calculate the actual dollar value of different improvement sizes
E-commerce example:
Current conversion rate: 2%
Monthly revenue: $100,000
Monthly visitors: 50,000
Potential improvements:
5% relative lift: 2% → 2.1% = +$2,500/month = $30,000/year
10% relative lift: 2% → 2.2% = +$5,000/month = $60,000/year
20% relative lift: 2% → 2.4% = +$10,000/month = $120,000/year
Step 2: Calculate the time cost of detecting each improvement
5% MDE (to detect $30,000/year improvement):
Required sample size: 120,000 per variation
Timeline: 240 weeks at 1,000 visitors/week
Result: 4.5 years to maybe find $30,000
20% MDE (to detect $120,000/year improvement):
Required sample size: 1,200 per variation
Timeline: 2.4 weeks at 1,000 visitors/week
Result: 3 weeks to maybe find $120,000
Step 3: Factor in opportunity cost
The 5% MDE approach:
4.5 years for one test = $30,000 maximum upside
Opportunity cost: 75+ other tests you could have run
Total potential: $30,000 (if you're lucky)
The 20% MDE approach:
3 weeks per test = 17 tests per year
Even with 50% failure rate: 8-9 winning tests
Total potential: $960,000+ in improvements
Step 4: Ask the right stakeholder question
Wrong question: "Should we test for 5% or 20% improvements?" Right question: "Should we spend 4 years hunting for $30,000, or 3 weeks hunting for $120,000?"
Why most companies miss this: They focus on conversion rate percentages instead of tying tests to actual monetary value pre-test. The better approach is calculating dollar impact first, then determining which MDE makes business sense.
The #1 Mistake: How Most Teams Set MDE Backwards (And Kill Their Testing Programs)
The wrong way (what 90% of teams do):
Open testing tool
See "Minimum Detectable Effect" field
Think "smaller sounds better" or pick a larger MDE because it needs less traffic
Enter a percentage without understanding what the percentage means
Wonder why tests take 6 months and never reach significance, running tests forever
What's actually happening: You just committed to needing 100,000+ visitors per variation to detect a 5% improvement. If you get 1,000 visitors per week, that's a 2-year test.
The devastating cost: While you wait 2 years to detect a 5% improvement worth $30,000 annually, your competitor finds a 20% improvement worth $200,000 in 6 weeks and implements 15 more tests.
Why Teams Make This Mistake
Psychological bias: "5% sounds more precise and scientific than 20%" Reality: 5% MDE isn't more accurate—it's just a longer, more expensive test
Misunderstanding: "We want to catch small improvements" Reality: You want to catch profitable improvements fast enough to matter
False assumption: "Lower MDE = better testing" Reality: Lower MDE = slower testing that often produces no actionable results
The Real Cost of Getting MDE Wrong
Scenario: E-commerce site with 2% conversion rate, 50,000 monthly visitors
Team picks 5% MDE (sounds precise):
Test duration: 8+ months
Improvements tested per year: 1-2
Revenue impact discovered: Maybe $30,000 if the test works
Team picks 20% MDE (sounds less precise):
Test duration: 3-4 weeks
Improvements tested per year: 15-20
Revenue impact discovered: $200,000+ from multiple wins
The paradox: The "less precise" approach delivers 6x more revenue because you actually finish tests and find improvements.
What is Minimum Detectable Effect (MDE)?
Simple definition: MDE is the smallest improvement your test can reliably detect.
Think of it like a telescope. A basic telescope might only detect stars brighter than magnitude 3. A powerful telescope can detect stars as faint as magnitude 9. But the powerful telescope takes much longer to set up and costs more.
MDE works the same way:
High MDE (20%): Quick, cheap tests that only catch big improvements
Low MDE (5%): Expensive, longer-running tests that catch tiny improvements
The key insight: You're not choosing how much improvement you'll get. You're choosing the smallest improvement you want your test to be able to detect.
Your MDE choice determines everything: test duration, required traffic, and which improvements you'll catch.
Examples of Getting This Right
Case Study 1: SaaS Startup
Situation: 500 weekly visitors, 1% trial conversion
Temptation: Set 5% MDE to be "thorough"
Reality check: Would take 10+ years for results
Smart choice: 25% MDE, 4-week tests, found 3 major improvements in first quarter worth $180,000 annually
Case Study 2: E-commerce Scale-up
Situation: 5,000 weekly visitors, 3% purchase rate
Previous approach: 8% MDE, tests took 6 months, team got frustrated
New approach: 15% MDE, 4-week tests, team momentum improved
Result: 12 tests completed in year one vs. 2 tests previously
Case Study 3: Enterprise Company
Situation: 50,000 weekly visitors, mature optimization program
Approach: Tiered MDE strategy
New features: 20% MDE (need big wins to justify development)
Copy/design tweaks: 10% MDE (lower implementation cost)
Checkout optimization: 8% MDE (high-value area worth longer tests)
Result: Balanced speed and precision based on business context
Why MDE Determines Your Test Timeline
The relationship most people miss: Lower MDE = exponentially longer tests.
Here's what this looks like with a 2% baseline conversion rate:
50% relative MDE
Target: 2% → 3%
Sample size: 1,200 per variation
Timeline: 2.4 weeks (at 1,000 visitors/week)
25% relative MDE
Target: 2% → 2.5%
Sample size: 4,500 per variation
Timeline: 9 weeks (at 1,000 visitors/week)
10% relative MDE
Target: 2% → 2.2%
Sample size: 30,000 per variation
Timeline: 60 weeks (at 1,000 visitors/week)
5% relative MDE
Target: 2% → 2.1%
Sample size: 120,000 per variation
Timeline: 240 weeks (at 1,000 visitors/week)
The pattern: Halving your MDE roughly quadruples your required sample size.
This is why randomly picking "small" MDE values kills testing programs. A 5% MDE might sound better than 20%, but it could mean waiting 5 years for results instead of 5 weeks.
The Easy 3-Step Framework for Choosing MDE
Step 1: Calculate Business Impact First
Don't start with statistical preferences. Start with business value.
Framework questions:
What's the annual revenue impact of a 5%, 10%, 15%, and 20% improvement?
How long can you afford to wait for each result?
How many other tests could you run in that time?
What's the cumulative impact of running 10 quick tests vs. 1 slow test?
Step 2: Match MDE to Implementation Effort
Low-effort changes (copy, colors, button text):
Development time: Hours to days
Can afford longer tests for smaller improvements
MDE of 8-12% might make sense
Logic: Low opportunity cost if improvement is small
Medium-effort changes (layout changes, new features):
Development time: Weeks
Need moderate improvements to justify effort
MDE of 12-18% makes sense
Logic: Balance between thoroughness and reasonable ROI threshold
High-effort changes (new checkout flow, major redesigns):
Development time: Months
Need bigger improvements to justify development time
MDE of 18-25% makes more sense
Logic: High opportunity cost requires high confidence in meaningful impact
Step 3: Consider Your Testing Maturity and Traffic
New testing programs: Start with 18-25% MDE
Build confidence and skills quickly
Catch the "low-hanging fruit" first
Establish testing culture with fast wins
Rationale: You need momentum more than precision
Mature optimization programs: Move to 8-15% MDE
Already optimized obvious improvements
Higher traffic volumes support longer tests
Incremental gains become more valuable
Rationale: You've earned the right to hunt for smaller improvements
Traffic-based guidelines:
High-traffic sites (10,000+ weekly visitors): Can afford 8-12% MDE
Medium-traffic sites (1,000-10,000 weekly): Stick to 12-18% MDE
Low-traffic sites (<1,000 weekly): Use 20-25% MDE or focus on traffic growth first
Common MDE Mistakes (And How to Fix Them)
Mistake #1: "Let's detect 5% improvements because smaller is better"
Why this fails: A 5% MDE might require 6-12 months of testing Better approach: "Let's start with 18% MDE to get results in 4-6 weeks, then optimize further if we find big wins" How to explain to stakeholders: "We can spend 6 months looking for a $30,000 improvement, or 6 weeks looking for a $120,000 improvement. Which sounds like better use of our time?"
Mistake #2: Using the same MDE for every test
Why this fails: Different pages have different baselines and business impact Better approach: Adjust MDE based on baseline conversion rate and strategic importance
Example:
Homepage (0.5% baseline): 22% MDE = 0.5% → 0.61% (still significant traffic impact)
Checkout page (15% baseline): 12% MDE = 15% → 16.8% (high-value optimization area)
Both tests might have similar business impact despite different MDE settings.
Mistake #3: Not explaining what MDE actually means
What stakeholders hear: "We're only testing for 15% improvements" What you actually mean: "We can reliably detect improvements of 15% or larger" Better explanation: "Our test will catch any improvement of 15% or more. If the real improvement is 25%, we'll definitely see it. If it's 8%, we might miss it—but detecting 8% improvements would require 4x more testing time."
Easy MDE Selection by Test Type
Landing Page Tests
Typical baseline: 1-5%
Recommended MDE: 18-25%
Why: Lower baselines need larger relative improvements for meaningful business impact
Quick decision: Use 20% MDE unless you have 20,000+ weekly visitors
Email Campaign Tests
Typical baseline: 15-30%
Recommended MDE: 12-18%
Why: Higher baselines make smaller relative improvements more valuable
Quick decision: Use 15% MDE for most email tests
Checkout Optimization
Typical baseline: 10-50%
Recommended MDE: 8-15%
Why: High-value area justifies longer tests for smaller improvements
Quick decision: Use 10% MDE (this is your highest-value testing area)
Product Page Tests
Typical baseline: 2-8%
Recommended MDE: 15-22%
Why: Balance between business impact and reasonable test duration
Quick decision: Use 18% MDE unless traffic is very high
How to Communicate MDE to Stakeholders
Use Business Language, Not Statistical Jargon
Don't say: "We're setting alpha at 0.05 and beta at 0.2 with 15% MDE" Do say: "We'll run this test for 4 weeks. If there's a 15% improvement or better, we'll definitely catch it. Smaller improvements would require 4+ months of testing."
Frame It as Smart Resource Allocation
When stakeholders ask: "What if there's only a 10% improvement and we miss it?" Your answer: "Then we move to the next test and potentially find a 25% improvement instead. Missing a 10% improvement costs us less than spending 4 months to find it."
Use Concrete Revenue Numbers
Instead of percentages, use dollars:
Before: "We're testing for 15% relative improvement" After: "We're testing to see if we can increase monthly revenue from $100,000 to $115,000, which would add $180,000 annually"
Quick MDE Decision Framework
Ask these three questions:
Business Impact: What annual revenue increase would justify this test effort?
Implementation Cost: How much development/design work is required?
Testing Timeline: How quickly do you need results?
High impact + Low cost + Flexible timeline = Lower MDE (8-12%) Medium impact + High cost + Tight timeline = Higher MDE (20-25%)
What to Do When Results Don't Meet Your MDE
Scenario: You set 15% MDE but only found 10% improvement
Don't say: "The test failed" or "Results aren't significant" Do say: "We found a 10% improvement, but our test wasn't large enough to be confident it's real. We have three options."
Your options:
Extend the test to detect the smaller effect (if timeline allows)
Implement based on directional evidence (if downside risk is low)
Move to the next test (if opportunity cost is high)
Decision framework: Compare the potential value of a 10% improvement vs. the cost of delaying your next test by 2-3 months.
Advanced Section: Understanding Relative vs Absolute Improvements
Here's where most people get confused: A/B testing tools typically use relative lift for MDE settings.
Absolute Lift
What it is: The raw percentage point difference Example: 2.0% → 2.2% = 0.2 percentage point absolute lift
Relative Lift
What it is: The percentage change from baseline Example: 2.0% → 2.2% = 10% relative lift (because 0.2 ÷ 2.0 = 0.10 = 10%)
Critical insight: The same relative MDE produces very different absolute improvements depending on your baseline.
Real Examples That Show the Difference
E-commerce Product Page
Baseline conversion: 3%
MDE setting: 15% relative improvement
What you're testing for: 3% → 3.45% (0.45 percentage point absolute lift)
Email Newsletter Signup
Baseline conversion: 15%
MDE setting: 15% relative improvement
What you're testing for: 15% → 17.25% (2.25 percentage point absolute lift)
SaaS Free Trial Conversion
Baseline conversion: 0.8%
MDE setting: 15% relative improvement
What you're testing for: 0.8% → 0.92% (0.12 percentage point absolute lift)
Notice: Same 15% relative MDE, completely different business impact. The email signup test detects a 2.25 point improvement while the SaaS test only detects 0.12 points.
Your Easy MDE Action Plan
This Week:
Audit current tests: What MDE are you using and why?
Calculate baseline rates for your top conversion pages
Estimate business impact of 10%, 15%, and 20% improvements
Choose MDE based on impact vs timeline tradeoffs
This Month:
Create MDE templates for different test types
Train stakeholders on the business impact framework
Track prediction accuracy: Are your MDE choices producing actionable results?
Long-term:
Optimize your strategy based on testing maturity and traffic growth
Document your MDE decision process for consistency
Refine your framework based on what you learn
Key Takeaways
MDE isn't about statistical perfectionism—it's about making smart business tradeoffs between test duration and insight quality.
Start with higher MDE values (18-22%) to build testing momentum, then lower them as your program matures and traffic grows.
The goal is consistent, actionable testing, not perfect statistical precision.
Most companies never master MDE selection, which gives you a massive competitive advantage in both optimization speed and quality. Get this right, and you'll run better tests faster than 90% of your competitors.