A/B Test Sample Size Calculator

For comparing two proportions, the per-arm sample size is approximately n = (z₁₋α/2 + z₁₋β)² × [p₁(1−p₁) + p₂(1−p₂)] ÷ (p₂ − p₁)², where p₁ is the baseline rate and p₂ = p₁ × (1 + MDE). At 95% confidence and 80% power the z-values are 1.96 and 0.84, so the leading constant is about 7.85. The denominator is the squared absolute difference — which is why halving the effect you want to detect roughly quadruples the required sample.

Baseline rate matters too: lower conversion rates need more traffic for the same relative effect, because each conversion carries more of the signal. A 2% baseline needs roughly twice the visitors of a 4% baseline to detect the same relative lift. This is a quiet argument for testing on higher-converting surfaces and segments when you have the choice.

The MDE is a business decision disguised as a statistics input: it should be the smallest lift that justifies the cost of running and shipping the change, not the lift you hope for. Setting MDE at an optimistic 25% makes tests short but blind to real 10% improvements — they’ll come back “not significant” and get wrongly discarded. For app store creative tests, 10–20% relative MDE is a pragmatic band; below 10% is usually only feasible for high-traffic apps.

Once the calculator gives visitors per arm, duration = required visitors ÷ daily traffic per arm — and you should run whole weeks even if you hit the number mid-week, because store traffic composition shifts across weekdays and weekends. If the implied duration exceeds 6–8 weeks, don’t start that test: raise the MDE, test a bolder change, or test on a higher-traffic surface. A test that drags for a quarter blocks the experiment slot and often outlives the seasonality that made its data comparable.