A/B Test Significance Calculator

What does this calculator do?

Enter the number of visitors and conversions for a control (A) and a variant (B), and this tool runs a two-proportion z-test — the standard statistical test for comparing two conversion rates. It reports the conversion rate for each side, the absolute and relative difference, a z-score and p-value, a confidence interval for both the difference and the relative uplift, and a plain-language verdict on whether the result is statistically significant.

Everything is a two-proportion z-test under the hood: the pooled conversion rate and standard error are used to compute the z-score and p-value (the hypothesis test), while an unpooled standard error is used for the confidence interval — the standard convention, since the interval shouldn't assume the two rates are equal the way the null hypothesis does.

How to use it

1. Enter visitors and conversions for Control (A) and Variant (B).
2. Pick a confidence level — 90%, 95% (the common default) or 99%.
3. Choose two-tailed if you just want to know whether B differs from A in either direction, or one-tailed if you specifically want to know whether B beats A.
4. Read the result table: the verdict row tells you in plain language whether to trust the difference, and the rows below it show the numbers behind that call.

What does "statistically significant" mean here?

It means the p-value is below your chosen significance threshold (1 − confidence level, e.g. 0.05 at 95% confidence) — in other words, if there were truly no difference between A and B, a gap this large would be unlikely to show up by chance alone. It is not a guarantee that B is actually better in the long run, and it says nothing about how big the improvement is — always read the uplift and its confidence interval alongside the p-value.

What sample size do I need?

There's no fixed number — it depends on your baseline conversion rate and the size of the effect you're trying to detect. As a rule of thumb, a combined sample under ~1,000 visitors is rarely enough to detect anything but a very large lift; smaller true differences need proportionally larger samples. If the calculator says "not enough data," keep the test running rather than stopping early — peeking at results repeatedly and stopping as soon as you see significance inflates your false-positive rate.

One-tailed vs. two-tailed — which should I pick?

Use two-tailed by default: it tests whether B is different from A, in either direction, and is the safer choice since a variant can always underperform. Use one-tailed only when you have a directional hypothesis decided before looking at the data (e.g. "B can only be better, never worse") — one-tailed tests reach significance with less evidence, so switching to one-tailed after seeing the data is a common way to fool yourself.

Why does the confidence interval matter as much as the p-value?

A p-value only tells you whether a difference is likely real; it says nothing about its size. The confidence interval for the relative uplift shows the plausible range of the true effect — a "significant" result with a CI of +2% to +54% is far less actionable than one with a CI of +22% to +34%, even if both clear the same p-value bar.

Privacy

All calculations run locally in your browser — your numbers are never sent to a server.