[4/5] Testing Fundementals: Three Test Types You Need to Know
Match the right test type to your situation
Three Test Types: A/B, Multivariate, and Bandit
⏱️ 2-min read
Not every testing situation calls for a simple A/B split. Sometimes you need different approaches. Here’s when to use each type.
A/B Tests: The Foundation
A/B tests compare two versions: your original versus one new idea. You send half your traffic to each version and measure which converts better.
Use A/B tests for:
Testing big changes like new layouts
Different value propositions
Redesigned checkout flows
Major copy rewrites
Traffic needed: Moderate. A few thousand visitors per version usually works.
Setup time: Fast. Most A/B tests take 30 minutes to two hours to build.
Example: Your pricing page has a long form with 12 fields. Users abandon it constantly. You create a variation with just 4 required fields. You test the long form versus the short form. Whichever converts more people wins.
A/B/n Tests: Multiple Variations
A/B/n tests compare your original against multiple variations at once. You might test A vs B vs C vs D, splitting traffic equally among all versions.
Use A/B/n tests for:
Testing different approaches to the same hypothesis
Three or four different headline angles
Multiple product images
Various call to action options
Traffic needed: High. Each additional variation requires more visitors. A four way test needs roughly twice as much traffic as a simple A/B test.
Setup time: Medium. Building multiple variations takes longer.
Example: You test four different email signup headlines. You split traffic 25% to each version. After two weeks, one version converts significantly better. You implement it.
The benefit is efficiency. You test multiple ideas in one experiment instead of running three separate A/B tests one after another. The downside is you need substantial traffic.
Multivariate Tests: Testing Combinations
Multivariate tests examine multiple elements and their combinations simultaneously. Instead of testing entire page versions, you test specific elements interacting together.
Imagine testing three things on one page:
Headline (2 options)
Hero image (2 options)
Call to action button (2 options)
A multivariate test creates all 8 possible combinations and tests them all at once. It tells you which combination wins and which individual elements matter most.
Use multivariate tests for:
Fine tuning pages that already convert well
Understanding how elements interact
Polishing high traffic pages
Squeezing extra percentage points
Traffic needed: Massive. You need 10,000+ visitors minimum.
Setup time: Long. Multivariate tests are complex to build and analyze.
Most companies should run 10 A/B tests for every 1 multivariate test. A/B tests give you bigger wins because you’re testing larger changes.
Bandit Tests: Adaptive Testing
Bandit tests start like A/B tests but adapt in real time based on performance. The algorithm sends more traffic to whichever version is winning, constantly adjusting the split as data comes in.
A traditional A/B test keeps the 50/50 split throughout. A bandit test might start 50/50 but shift to 70/30, then 85/15, and eventually 95/5 as the winner becomes clear.
Use bandit tests for:
Short term campaigns with limited shelf life
Situations where minimizing lost conversions matters
Email subject line testing across a campaign
Promotional periods
Traffic needed: Moderate to high.
Setup time: Fast if your tool supports bandits.
Example: You’re running a two week promotional campaign with three different email subject lines. A bandit test automatically sends more traffic to whichever subject line gets better open rates. By day 3, the best subject line gets 80% of your remaining sends.
The downside of bandits is you learn less. Traditional A/B tests give you clean comparisons. Bandits optimize for short term wins at the cost of statistical clarity.
Which One Should You Use?
Start with A/B tests. They’re simple, reliable, and work for 90% of situations.
Use A/B/n when you have good traffic and want to test several variations of the same idea efficiently.
Save multivariate for when you have enormous traffic and want to fine tune.
Try bandits for time limited campaigns where you want to maximize revenue during testing.
━━━━━━━━━━━━━━━━━━━━━━
💡 QUICK WIN
For your next three tests, stick with simple A/B splits. Master the basics before trying fancy techniques. Simple tests teach you faster.
━━━━━━━━━━━━━━━━━━━━━━
Coming up in Part 5:
Five testing mistakes that waste traffic and make you look inexperienced.
Reply with questions anytime.
– Atticus
P.S. If someone tells you to run a multivariate test and your page gets less than 50,000 visitors per month, politely explain that you need way more traffic for that approach to work. Suggest A/B testing instead.