A/B Test Significance Calculator

The test asks: if A and B truly converted at the same underlying rate, how surprising is the difference we observed? It pools both arms into one estimated rate p̂ = (conversions A + conversions B) ÷ (visitors A + visitors B), computes the standard error of the difference SE = √(p̂(1−p̂)(1/nA + 1/nB)), and forms z = (rate B − rate A) ÷ SE. The z-score maps to a p-value via the normal distribution: |z| ≥ 1.96 corresponds to p ≤ 0.05 two-tailed, the conventional significance threshold.

The normal approximation is reliable when each arm has a reasonable number of both conversions and non-conversions — a common rule of thumb is at least 10 of each per arm. App store tests usually clear that easily on impressions, but keyword-level or country-level slices may not; below those counts, treat the p-value as indicative rather than exact.

The most damaging habit in conversion testing is peeking: checking significance daily and stopping the moment p dips under 0.05. Because random fluctuation crosses the threshold transiently, this inflates your false-positive rate far above the nominal 5%. The fix is to fix the sample size in advance (see the sample size calculator), run until you reach it, and test once. Also run tests over whole weeks — app store traffic and conversion swing by day of week, and a weekday-vs-weekend imbalance between arms masquerades as a treatment effect.

Significant is not the same as important. With enough traffic, a 0.5% relative lift becomes statistically significant while being operationally irrelevant — and conversely, PPO tests on low-traffic apps may never reach significance for effects under ~10%. Decide your minimum effect worth shipping before the test, and remember Apple’s PPO applies treatments to product page traffic while much of your install volume converts directly from search results, so the store-wide impact of a page-level winner is usually smaller than the test’s measured lift.