· 6 min read
Every ad platform shows you the winners
Abraham Wald worked out where to armour bombers by thinking about the ones that never came back. Advertising has the same blind spot, and almost every case study you have read is built on it.
In 1943 the American military had a problem with bombers. Too many were not coming back. They examined the ones that did, mapped the bullet holes, and found the damage clustered on the wings, the fuselage, the tail gunner's position. The obvious move was to armour where the holes were.
Abraham Wald, a statistician at Columbia, said the opposite. Armour where there are no holes.
The planes he was looking at were the survivors. Every one of them had been hit and made it home, which meant hits in those places were survivable by definition. The engines were unmarked in his sample not because they never got hit, but because a plane hit there did not come back to be measured. The data set had a hole in it shaped exactly like the answer.
This is called survivorship bias, and once you see it you cannot stop seeing it. It is also, more or less, the business model of digital advertising.
The case study is the returning bomber
Open any ad platform's customer page. Every logo on it is a plane that made it home.
"How Acme cut CAC by 60%." "How a two-person team hit $40k MRR from paid." All true, presumably. All measured on survivors.
The companies that spent the same money, ran the same playbook, and got nothing are not on that page. They are not on any page. They churned quietly, told nobody, and their absence is the most informative thing about the whole exercise — and it is invisible.
You cannot compute a success rate from a wall of successes. But that wall is what almost every advertising decision gets made against.
It gets worse, because the selection happens twice. First, only the winners get written up. Second, the winners self-select into being written up — the ones who did well are delighted to be a case study, the ones who did badly stop answering emails. The sample is filtered, then filtered again by the same criterion.
Three places this bites in advertising
Benchmarks. An average conversion rate of 8.18% sounds like a target to hit. It is an average over accounts that were still running when the data was collected. Accounts that failed and were switched off in month two are, by construction, less likely to be in it. Real benchmarks are honest about their method — WordStream's says plainly it covers 13,000+ live campaigns — but "live campaigns" is the filter. The dead ones had numbers too.
Channel folklore. "Reddit ads work great for developer tools." Maybe. Or: the people for whom Reddit worked wrote threads about it, and the people for whom it didn't wrote nothing, and you are reading the output of that filter and calling it evidence.
Your own history. The campaign you remember as a triumph is the one you kept running. The three you killed in week one are not in your mental average. Your intuition about what works is trained on a survivor sample too.
What honest measurement would look like
The fix is not more data. It is data that includes the failures.
That means publishing losses at the same volume and prominence as wins. It means a denominator: not "look at this brand's results" but "here is what every brand got, including the ones that got nothing." It means the reader being able to compute the failure rate themselves rather than taking a filtered highlight reel on trust.
Almost no advertising platform does this, and the reason is not conspiracy. It's that a platform's incentive runs the other way. The failures are bad for business, and nobody is forcing anybody to publish them.
What we did about it
I should be upfront: I run an ad board, so this is where I tell you what we built. Take it as a worked example of the principle rather than a pitch.
BidSurvivor auctions advertising an hour at a time. The highest bid holds the hour and can be outbid by anyone at any time. Two things follow from that structure, and both of them are about the bombers that didn't come home.
Losing is public. The ticker across the bottom of the board reads, in so many words, "XYZ couldn't survive the bid of ABC." Being displaced is not hidden or softened. It scrolls past in front of everyone. The site is called BidSurvivor because most bids do not survive, and that is the honest description of an auction.
Every brand's numbers are public, not just the good ones. Every brand that has held an hour has a page showing what the board actually sent it — card opens, click-throughs, the rate between them. Not a selected sample. All of them, including the ones where the answer is embarrassing.
One detail we argued about and got right, I think: we withhold the click-through rate until a brand has at least five card opens. Two visits from two opens is not "100%". It is not enough to divide by, and a headline number that means nothing is worse than no number. Small samples are how survivorship bias sneaks back in through the front door.
That means you can look at the board and conclude a slot is not worth your money. We would rather you did that than bought one on a promise and left.
The uncomfortable part
Wald's insight only works if you are willing to reason about data you cannot see. That is harder than it sounds, because the missing data is missing — it does not announce itself, it does not show up as an error, it just quietly is not there.
The habit worth building is small. Whenever you see an advertising result, ask one question: what would the ones that failed have looked like, and would I ever have found out?
If the answer is "I would never have found out", you are looking at the wings of a bomber and about to armour the wrong part of the plane.
The Wald story is well documented; his original wartime memoranda were published by the Statistical Research Group at Columbia and are widely reproduced. The benchmark figures cited are from WordStream's 2026 Google Ads benchmarks.