How many clicks before a conversion rate means anything?

You spend forty dollars, get sixty clicks, and one of them signs up. Is 1.7% good? The honest answer is that you do not have a conversion rate yet — you have one signup and a lot of arithmetic. Sample size is the thing standing between a number and a decision, and at small-budget scale it is almost always the thing being ignored. (Disclosure: published by BidSurvivor, which sells advertising and therefore has an obvious interest in you buying some. The arithmetic below is not ours and does not care.)

Why one signup tells you almost nothing

Conversions are rare events. When something happens rarely, the count you observe swings wildly around the true rate even when nothing about the world has changed.

Take a page that genuinely converts at 3%. Send it 60 visitors. The most likely single outcome is 2 conversions — but 0 happens about 16% of the time, and 5 or more happens often enough that you will see it if you run the test a few times. So the same page, unchanged, will show you 0%, 3.3%, and 8.3% on three consecutive Tuesdays.

Now imagine you changed the headline between those Tuesdays. You will conclude that the headline worked. It did not. You measured noise and gave it a cause.

This is not a subtle statistical point. It is the single most common way small advertisers waste money: not on the ads, but on the confident wrong conclusions drawn from them, which then determine where the next few hundred dollars go.

The rule of thumb worth memorising

For a rough sanity check, you want at least 15 conversions before a rate is worth quoting, and considerably more before it is worth comparing against another rate.

Fifteen is not a magic number — it is roughly where the relative standard error of a count drops under 26%, which is the point where the number stops moving around so much that it tells you a different story each week. Below it, your rate is a rumour.

Turn that around and it becomes a planning tool. If you believe your page converts somewhere near 3%, then:

• 15 conversions ÷ 0.03 = 500 clicks before the rate means much
• At $0.60 a click, that is $300 to learn one number

At 1% it is 1,500 clicks. At 10% it is 150. The lower your conversion rate, the more expensive certainty gets — which is exactly backwards from what most people assume, because a low rate feels like it should be cheap to establish.

This is the calculation behind the confidence verdict in our ad budget calculator: it tells you not just what a budget buys, but whether it buys enough clicks for the resulting number to be worth anything. Most calculators stop before that second part, which is the part that decides whether you should trust the first.

Comparing two things is much harder than measuring one

Everything above is for measuring a single rate. Comparing two — an A/B test — is a different and much more expensive problem, because now the noise in both numbers has to be smaller than the difference between them.

The practical consequence: detecting a change from 3% to 3.6% (a 20% relative improvement, which would be a very good result) needs somewhere in the region of 13,000 visitors per variant. Evan Miller's sample size calculator will do the exact arithmetic for whatever numbers you actually have, and it is worth ten minutes with your own figures before you plan a test rather than after.

For most people reading this, that number ends the conversation. If you are spending $200 a month, you cannot run a valid A/B test on conversion rate. Not "it will take a while" — you cannot. The budget required to detect the size of improvement you are hoping for exceeds the budget you are optimising.

Miller's related piece, How Not To Run An A/B Test, covers the other half of the trap: watching a test and stopping it the moment it looks significant. Do that and you will find a "winner" almost every time, including between two identical pages.

What to do instead when you cannot afford a sample size

Refusing to measure is not the answer either. Three things work at small scale:

Measure further up the funnel. Clicks are 30 times more common than conversions at a 3% rate, so click-through rate reaches a usable sample size 30 times faster. You can honestly compare two ad creatives on CTR with a few hundred impressions each. You cannot compare them on signups. Test the thing you have enough data for, and be explicit that you are testing the proxy.

Look for differences big enough to see. Small effects need large samples; enormous ones do not. If a change takes you from 0 signups in 200 clicks to 11 in 200 clicks, you do not need a calculator. Small budgets should be hunting for order-of-magnitude differences — a different audience, a different channel, a different offer — not 15% lifts on a button colour.

Count the qualitative. Six people telling you the pricing page confused them is real evidence, gathered from a sample that no statistical test would accept for a rate. It is not a conversion rate and should not be reported as one, but it will improve the page more reliably than a test you cannot afford to run.

The sentence to keep

When someone shows you a conversion rate, ask what the denominator was. If the answer is under a few hundred, or the numerator is under fifteen, the number is a story rather than a measurement — and the more precisely it is quoted, the less it means. A "2.7% conversion rate" from 74 clicks and 2 signups is two signups.

This is also why honest numbers matter more than fast tests when the budget is small: a fast test that produces a confident wrong answer is worse than no test, because you will act on it. And it is why a $50 advertising budget is better spent buying one clear signal than split three ways into three ambiguous ones.

Spend enough to learn one thing properly, or spend nothing and go talk to six people. The middle — enough to produce numbers, not enough to trust them — is where most small advertising budgets die.