A/B Test Statistical Significance Without the Jargon
What a p-value and confidence level mean in plain language, how to plan sample size, and why peeking at results early gives false winners.
What significance is asking
An A/B test compares two versions, usually by conversion rate. The key question is whether the difference you see is real or could easily be random noise. Statistical significance is a way of answering that.
The p-value is the probability of seeing a difference at least this large if the two versions were actually the same. A small p-value means the result would be unusual by luck alone. The common threshold is 0.05, which corresponds to a 95 percent confidence level.
What a result does not mean
A p-value below 0.05 does not mean there is a 95 percent chance that version B is better. It does not tell you how big the improvement is either. And a significant result can still be tiny. With a very large sample, a 0.1 percent lift can be statistically significant and still not worth the cost of the change. Always look at the size of the effect as well as the p-value.
Decide the sample size first
The most reliable tests fix their sample size before they start. Three inputs decide it: your current conversion rate, the smallest improvement you care about detecting, and how much risk of error you accept. Smaller improvements need far larger samples. Detecting a lift from 5.0 to 5.1 percent takes many times more visitors than detecting a lift from 5 to 6 percent.
If your site cannot reach the needed sample in a reasonable time, test a bigger change rather than a tiny tweak.
Do not stop the test when it first looks good
Checking the result every day and stopping the moment it crosses the threshold is called peeking, and it inflates false positives badly. Early results swing a lot, and if you give yourself many chances to see a winner, you will eventually see one that is not real.
Set the end date or sample size up front and judge the result only then. Also run tests for at least one full week, and ideally two, so that weekday and weekend behavior are both included.
Practical rules
These habits prevent most bad decisions.
- Change one thing at a time so you know what caused the difference.
- Split traffic randomly and keep each visitor in the same version.
- Do not run the test during an unusual period, such as a big sale, unless the result is only meant for that period.
- Record the result either way. A test that finds no difference is still useful information.
- Treat a significant result as strong evidence, not as proof, and confirm important wins with a follow up test.