Every ad platform shows you the winners

The clearest illustration of survivorship bias in advertising was worked out in 1943, before it had anything to do with advertising. The American military had a problem with bombers. Too many were not coming back. They examined the ones that did, mapped the bullet holes, and found the damage clustered on the wings, the fuselage, the tail gunner's position. The obvious move was to armour where the holes were.

Abraham Wald, a statistician at Columbia, said the opposite. Armour where there are no holes.

The planes he was looking at were the survivors. Every one of them had been hit and made it home, which meant hits in those places were survivable by definition. The engines were unmarked in his sample not because they never got hit, but because a plane hit there did not come back to be measured. The data set had a hole in it shaped exactly like the answer.

This is called survivorship bias, and once you see it you cannot stop seeing it. It is also, more or less, the business model of digital advertising.

The case study is the returning bomber

Open any ad platform's customer page. Every logo on it is a plane that made it home.

"How Acme cut CAC by 60%." "How a two-person team hit $40k MRR from paid." All true, presumably. All measured on survivors.

The companies that spent the same money, ran the same playbook, and got nothing are not on that page. They are not on any page. They churned quietly, told nobody, and their absence is the most informative thing about the whole exercise — and it is invisible.

You cannot compute a success rate from a wall of successes. But that wall is what almost every advertising decision gets made against.

It gets worse, because the selection happens twice. First, only the winners get written up. Second, the winners self-select into being written up — the ones who did well are delighted to be a case study, the ones who did badly stop answering emails. The sample is filtered, then filtered again by the same criterion.

Three places this bites in advertising

Benchmarks. An average conversion rate of 8.18% sounds like a target to hit. It is an average over accounts that were still running when the data was collected. Accounts that failed and were switched off in month two are, by construction, less likely to be in it. Real benchmarks are honest about their method — WordStream's says plainly it covers 13,000+ live campaigns — but "live campaigns" is the filter. The dead ones had numbers too.

Channel folklore. "Reddit ads work great for developer tools." Maybe. Or: the people for whom Reddit worked wrote threads about it, and the people for whom it didn't wrote nothing, and you are reading the output of that filter and calling it evidence.

Your own history. The campaign you remember as a triumph is the one you kept running. The three you killed in week one are not in your mental average. Your intuition about what works is trained on a survivor sample too.

How to read a case study anyway

You will keep reading them, so here is the habit that makes them useful rather than misleading.

1. Ask for the denominator. How many customers ran this playbook? If the answer is "we don't publish that", the case study is an anecdote, and should be weighed like one.
2. Look for the metric that was not chosen. A case study that reports return on ad spend and not the spend, or clicks and not conversions, chose the number that survived. Vanity metrics vs real clicks is the field guide to that choice.
3. Find the date. A result from a platform two years ago is a result from a different auction with different prices.
4. Prefer sellers who publish everything. A short list of ad platforms that publish their numbers exists, and it is short because publishing losses is bad for business. The ad rate comparison shows what those platforms actually charge.

What honest measurement would look like

The fix is not more data. It is data that includes the failures.

That means publishing losses at the same volume and prominence as wins. It means a denominator: not "look at this brand's results" but "here is what every brand got, including the ones that got nothing." It means the reader being able to compute the failure rate themselves rather than taking a filtered highlight reel on trust.

Almost no advertising platform does this, and the reason is not conspiracy. It's that a platform's incentive runs the other way. The failures are bad for business, and nobody is forcing anybody to publish them.

What we did about it

I should be upfront: I run an ad board, so this is where I tell you what we built. Take it as a worked example of the principle rather than a pitch — the complete rules are on their own page.

BidSurvivor auctions advertising twelve hours at a time, two slots a day. The highest bid holds the slot and can be outbid by anyone at any time. Two things follow from that structure, and both of them are about the bombers that didn't come home.

Losing is public. The ticker across the bottom of the board reads, in so many words, "XYZ couldn't survive the bid of ABC." Being displaced is not hidden or softened. It scrolls past in front of everyone. The site is called BidSurvivor because most bids do not survive, and that is the honest description of an auction.

Every brand's numbers are public, not just the good ones. Every brand that has held a slot has a page showing what the board actually sent it — card opens, click-throughs, the rate between them. Not a selected sample. All of them, including the ones where the answer is embarrassing. The live stats page shows the aggregate totals.

One detail we argued about and got right, I think: we withhold the click-through rate until a brand has at least five card opens. Two visits from two opens is not "100%". It is not enough to divide by, and a headline number that means nothing is worse than no number. Small samples are how survivorship bias sneaks back in through the front door.

That means you can look at the board and conclude a slot is not worth your money. We would rather you did that than bought one on a promise and left. It also means the board is only useful as a fast test if the readout is honest, which is the argument of fast marketing tests need honest numbers.

The uncomfortable part

Wald's insight only works if you are willing to reason about data you cannot see. That is harder than it sounds, because the missing data is missing — it does not announce itself, it does not show up as an error, it just quietly is not there.

The habit worth building is small. Whenever you see an advertising result, ask one question: what would the ones that failed have looked like, and would I ever have found out?

If the answer is "I would never have found out", you are looking at the wings of a bomber and about to armour the wrong part of the plane.

---

The Wald story is well documented; his original wartime memoranda were published by the Statistical Research Group at Columbia and are widely reproduced. The benchmark figures cited are from WordStream's 2026 Google Ads benchmarks.