A/B Test Statistical Significance Calculator

データ開発者数学

[iotools_ab_test_significance_calculator]

ガイド

A/B Test Statistical Significance Calculator

Drop in the visitors and conversions for your Control (A) and Variant (B) and instantly see whether the difference between them is real or just noise. The calculator runs a two-proportion z-test, computes the p-value, the absolute and relative lift, and the confidence interval for the difference, then turns the numbers into a clear verdict: winner, no difference, or keep the test running.

It uses the same pooled-variance z-test that powers the significance calculators behind tools like Optimizely, VWO, and Convert. All math runs in the browser, so your experiment numbers never leave the page.

使用方法

Enter the total number of visitors who saw the Control (A) experience and how many of them converted.
Enter the total number of visitors who saw the Variant (B) experience and how many of them converted.
Pick a confidence level (90%, 95%, or 99%) and choose a one-tailed or two-tailed test.
Read the verdict at the top of the results panel and check the breakdown for the p-value, z-score, lift, and confidence interval.
Click any of the example presets at the bottom of the inputs to see how a clear winner, a borderline result, and a low-traffic test look.

機能

Two-proportion z-test — pooled-variance significance test, the standard for comparing conversion rates.
Exact p-value — computed from the standard normal CDF using a high-precision erf approximation.
One-tailed or two-tailed — choose based on whether you only care about B beating A or about any difference.
Confidence intervals — both for the absolute difference (in percentage points) and for the relative lift.
Smart verdicts — distinguishes between not enough data, no real difference, and a difference that needs more samples.
Visual breakdown — bars for conversion rates and a traffic-split overview so you can spot imbalanced allocations.
100%クライアントサイド — your experiment numbers stay in your browser, nothing is uploaded.

広告削除する？

 よくある質問

What is statistical significance in an A/B test?

Statistical significance is the probability that the difference observed between two variants is caused by a real effect rather than random chance. The conventional threshold is a p-value below 0.05, meaning there is less than a 5% chance the observed difference would appear if the variants were actually identical.
What does the p-value actually mean?

The p-value is the probability of seeing a difference as large as the one you observed (or larger) if the two variants truly performed the same. A small p-value (for example, 0.01) means that result would be very unlikely under the null hypothesis, which is evidence the variants behave differently. The p-value is not the probability that B is better than A.
When should I use a one-tailed vs two-tailed test?

Use a two-tailed test when you want to detect any difference between A and B, including B performing worse. Use a one-tailed test when you have a directional hypothesis (B is better than A) decided before running the experiment and you would not act on a result in the opposite direction. Two-tailed is the safer default for most product experiments.
What is the difference between absolute lift and relative lift?

Absolute lift is the raw difference in conversion rates expressed in percentage points (for example, going from 5% to 6% is a +1 percentage point absolute lift). Relative lift expresses that change as a percentage of the original rate (going from 5% to 6% is a +20% relative lift). Relative lift is the number marketers usually quote, but absolute lift is what determines the dollar impact at a given traffic volume.