In A/B test design, what does statistical significance indicate?

• A. A/B testing is simply comparing two campaigns with equal samples, significance is not needed.
• B. A/B testing is a controlled experiment with a treatment and control; significance measures whether observed difference is unlikely due to chance, typically p<0.05.
• C. A/B testing uses multiple variants but not experimental control.
• D. A/B testing is only about design aesthetics.

Answer: (B)

In A/B testing, statistical significance shows whether the observed difference between the treatment and the control is likely real and not just random noise from sampling. When you randomly assign users to the two groups, you’re trying to isolate the effect of the change. If the result is statistically significant (commonly with a p-value below a threshold like 0.05), it means that the probability of seeing such a difference if there were no true effect is very low, so you have evidence that the treatment had an effect.

But significance doesn’t guarantee practical importance. A result can be statistically significant yet have a tiny effect size that doesn’t matter in business terms, especially if the sample is large. That’s why it’s important to also consider the size of the effect, confidence intervals, and whether the experiment is adequately powered.

Other statements don’t fit because A/B testing is a controlled, randomized comparison designed to measure the effect of a change, and significance is about confirming that observed differences are unlikely due to chance, not about aesthetics or about evaluating multiple variants without a proper control.