[3/5] Testing Fundamentals: The 6-Step Testing Process
The repeatable process pros use every time
The 6-Step Testing Process
⏱️ 2-min read
Good testers follow a process. They don’t wing it. Here’s the exact process that works every time.
Step 1: Pick One Thing to Test
Your homepage headline. Your call to action button. Your product image. Just one variable at a time.
Don’t test everything at once. Don’t test random things because a blog post said button color matters. Pick something that research tells you is causing problems.
Where does research come from? Analytics showing where people drop off. Session recordings showing where people get confused. Customer surveys revealing what concerns them.
Step 2: Create Your Variation
Make a single change. If you test your headline, keep everything else identical. Change too many things and you won’t know what caused the result.
Your variation should solve a specific problem you identified in Step 1. It should make sense. It should have a clear reason for existing.
Write down your hypothesis: “Changing X because Y will result in Z.” This matters for documentation later.
Step 3: Calculate Sample Size
Use a sample size calculator to determine how many visitors you need. Here’s what you input:
Your current conversion rate (the control’s baseline)
The minimum improvement you want to detect (usually 10% to 20%)
Your confidence level (use 95%)
Your statistical power (use 80%)
The calculator tells you how many visitors you need per variation before you can trust the results.
If the calculator says you need 5,000 visitors per variation and you only get 500 visitors per week to that page, your test will take 20 weeks. That’s too long. Pick a higher traffic page or test something with bigger expected impact.
Step 4: Let It Run
Most tests need at least two weeks and a few hundred conversions per version. Some need more time. Trust your sample size calculation from Step 3.
Do not peek at results every day and make decisions based on early trends. Early data is noisy. Results swing wildly in the first few days then stabilize.
Run tests for full weeks. If you start on Wednesday, end on Wednesday. This controls for day of week effects. Weekends might convert differently than weekdays.
Step 5: Check Statistical Significance
After your test reaches the predetermined sample size, check the results. Your testing tool shows you:
Conversion rate for control
Conversion rate for variation
Statistical significance level
Look for 95% confidence minimum. That means if you ran this exact test 100 times, about 95 of those times you’d see the same result.
For simplicity's sake, anything below 95% means you can’t trust the result. The difference might just be random chance.
Step 6: Learn Something
Whether you win or lose, you now know more about what your audience responds to.
If the variation won, implement it and test something else.
If the variation lost, your hypothesis was wrong. Think about why. Maybe you solved the wrong problem. Maybe your solution wasn’t strong enough. Test a different approach.
If results are flat (no real difference), the change doesn’t matter to your audience. Move on to testing something with bigger potential impact.
Document what you learned. Write down the hypothesis, the result, and what you’ll try next. This becomes your testing knowledge base.
━━━━━━━━━━━━━━━━━━━━━━
💡 QUICK WIN
Before you launch your next test, answer: “How many visitors do I need?” Use a sample size calculator. Example Statsig Sample Size Calculator If the answer is more visitors than you’ll get in 4 weeks, pick a different test.
━━━━━━━━━━━━━━━━━━━━━━
Coming up in Part 4:
Three test types (A/B, Multivariate, Bandit) and when to use each one.
Reply with questions anytime.
– Atticus
P.S. The most common mistake is stopping tests when they reach significance after 3 days. Don’t. Results regress to the mean. Let tests run for the predetermined time even if early results look amazing.