A/B Test Significance Calculator

What an A/B test significance calculator does

An A/B test significance calculator checks whether a difference between two versions of something — a webpage, an email, a checkout button — is a real effect or just random chance. It takes the number of visitors and the number of conversions for each version and returns a p-value, a z-score, and a confidence interval. A conversion is whatever action counts as success: a purchase, a signup, a click.

How to use the calculator

The calculator needs four numbers.

• Visitors (control): how many people saw version A, the original.
• Conversions (control): how many of them converted.
• Visitors (variation): how many people saw version B, the new version.
• Conversions (variation): how many of them converted.

Two more settings change how the result is judged.

• Confidence level sets how strict the test is. 95% is the common default. A higher level, like 99%, demands stronger evidence before calling a result significant.
• Hypothesis sets the direction of the question. A two-sided test asks "is B different from A, in either direction?" A one-sided test asks only "is B better than A?" A one-sided test should only be chosen before the data comes in, and only when a worse result for B would not change any decision.

The result updates as the numbers are entered. The inputs are stored in the page's web address, so a result can be bookmarked or shared as a link.

A/B test formula (two-proportion z-test)

The calculator uses a two-proportion z-test, a standard method taught in introductory statistics for comparing two rates.

Step 1. Conversion rate of each group.

$p_1 = \frac{x_1}{n_1}, \qquad p_2 = \frac{x_2}{n_2}$

Here $x$ is the number of conversions and $n$ is the number of visitors, for control (1) and variation (2).

Step 2. Pooled conversion rate. This step assumes, for the sake of the test, that both groups share one true underlying rate.

$\hat{p} = \frac{x_1 + x_2}{n_1 + n_2}$

Step 3. Standard error of the difference, built from the pooled rate.

$SE = \sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}$

Step 4. Z-score, the gap between the two rates measured in standard errors.

$z = \frac{p_2 - p_1}{SE}$

Step 5. P-value, taken from the standard normal distribution $\Phi$. For a two-sided test: $p = 2\left(1 - \Phi(|z|)\right)$. For a one-sided test: $p = 1 - \Phi(z)$.

Step 6. Verdict. The result counts as statistically significant when the p-value is smaller than $\alpha$, where $\alpha = 1 - \text{confidence level}$. At 95% confidence, $\alpha = 0.05$.

The calculator also reports a confidence interval for the size of the difference, $p_2 - p_1$. This interval uses a separate standard error, one that does not assume the two groups share a rate:

$SE_{unpooled} = \sqrt{\frac{p_1(1-p_1)}{n_1} + \frac{p_2(1-p_2)}{n_2}}$

The interval is $(p_2 - p_1) \pm z_c \times SE_{unpooled}$, where $z_c$ is 1.6449 at 90% confidence, 1.96 at 95%, and 2.5758 at 99%. If the interval does not contain zero, the difference is significant at that confidence level.

Worked example

The pooled rate is $(100+150)/(1000+1000) = 0.125$, or 12.5%. The pooled standard error is about 0.0148. The z-score is $(0.15 - 0.10) / 0.0148 \approx 3.38$.

For a two-sided test, this gives a p-value of about 0.00072. That is below the usual 0.05 threshold, so at 95% confidence the difference is statistically significant. If the two versions truly performed the same, a gap this large or larger would appear in about 7 out of every 10,000 tests by chance alone.

The absolute difference is 5.00 percentage points. The relative uplift is 50%, since B's rate is 1.5 times A's rate. The 95% confidence interval for the true difference is about 2.11% to 7.89%.

Reading the result correctly

A statistically significant result is not automatically a result worth acting on. A tiny, real lift might not be worth the cost of shipping it. Business judgment still matters after the math is done.

Checking the p-value every day and stopping the test the moment it crosses 0.05 raises the odds of a false positive. It is better to pick a sample size or a run length in advance and let the test finish before reading the result.

The normal approximation this test relies on works best when each group has at least about 10 conversions and 10 non-conversions. With smaller numbers, the result should be treated as a rough signal rather than a firm answer.

A 95% confidence level does not mean there is a 95% chance that B is truly better. It means that if the same test were repeated many times, this method would give a wrong significant result at most 5% of the time. That is a statement about the method, not about this one result.

Testing many metrics or many variants at once raises the chance that at least one comparison looks significant purely by luck. Choosing one metric before the test starts avoids this problem.

Frequently asked questions

What sample size does an A/B test need? Enough visitors that a meaningful lift would actually show up as significant. A rough floor is a few hundred conversions per version. A dedicated sample-size calculator, using the baseline rate and the smallest lift worth detecting, gives a firmer number before a test starts.

Should the test be one-sided or two-sided? Two-sided is the safer default, since it can flag B as either better or worse than A. One-sided is more sensitive to an improvement but cannot flag a worse result, so it should only be used when a worse B would not change any decision.

Why does the calculator report a confidence interval? A confidence interval shows the likely range of the true difference, not just a significant/not-significant label. A wide interval that spans zero usually means the test needs more data.

Is this a Bayesian calculator? No. It uses a frequentist z-test, the method most commonly taught and used for A/B testing. A Bayesian tool instead reports a probability that B beats A. Both are valid; they answer slightly different questions.

What if the calculator shows an error? An error appears if a conversion count is negative, if it exceeds the visitor count, or if the combined conversion rate across both groups is 0% or 100%, since the z-test formula cannot divide by a zero standard error in that case.

Sources

• Two-proportion Z-test — Wikipedia
• NIST/SEMATECH e-Handbook of Statistical Methods — Comparing two proportions
• Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (Cambridge, 2020)