Ad copy test: is the winner real?
Two versions of an ad, two click rates, one of them higher. This tells you whether the gap is large enough to act on, or the kind of difference that turns up between two identical ads by chance.
CTR, version A
2.00%
CTR, version B
2.90%
Relative difference
+45.0%
Chance it's noise (p)
0.066
Impressions needed per version
4,628
To detect this gap reliably (95% / 80% power)
What this means
Not settled. A gap this size turns up by chance 7% of the time; you need about 4,628 impressions per version to know.
How it works
A two-proportion z-test on the click rates, with the two versions' traffic pooled for the standard error. The p-value is the chance of a gap at least this large if the two versions were really the same.
The traffic needed is the per-version sample that detects the observed gap 80% of the time at 95% confidence — the usual convention, and the one to decide before the test starts rather than after.