The evidence you see is what survived.
Conclusions drawn only from the cases that made it through, such as active users, successful products, or returning planes, ignore everything that was filtered out on the way. They come out too optimistic, and they can credit the wrong cause.
Cicero tells of Diagoras of Melos being shown painted offerings from sailors who had survived shipwrecks, as proof that the gods protect those who pray. Diagoras replied that there were no paintings of the ones who drowned, and they were far more numerous.
The best-known modern case comes from the Second World War. In 1943 Abraham Wald, a statistician in Columbia University’s Statistical Research Group, wrote memos on estimating how vulnerable each part of an aircraft was from the damage on planes that came back. Popular retellings simplify his mathematics, but the insight holds: returning planes can only show where a plane can be hit and still return.
Any group you can observe today has already been filtered by whatever removed the rest. Conclusions drawn from it describe the filter as much as the group, and they lean optimistic, because the failures are exactly what is missing.
Damage on the planes that returned showed where a plane could be hit and still fly home. The places without holes were where the lost planes had been hit, so those were the parts to protect.
Metrics on active users describe the people who got past onboarding. Those who hit a wall left and stopped producing data, so the dashboard looks healthier than the product.
Recruiting from the current user base asks the people the product already works for. The reasons it fails live with people who churned or never signed up.
Studying market leaders for the patterns behind their success ignores the products that used the same patterns and disappeared. A trait the winners share is not proof that it caused the win.
Case studies, showcases, and founder advice come from the survivors. The failures that made the same choices are silent, so what looks like a recipe may be luck.
Average revenue per customer can rise over time simply because low-value customers leave. The line shows who remained, not how anyone changed.
Nassim Taleb’s name for the data that survivorship hides: the failed, the churned, the unpublished. Ask what the missing group would have shown before trusting the one in front of you.
Track cohorts from the start, keep churned accounts in the denominator, and interview people who quit or never finished. The missing group is usually where the answer is.
Models learn from what was published and kept. The failures nobody wrote about are missing from the training data too.
A model learns from what was published and stayed online. Abandoned products, failed launches, and negative results are underrepresented, so its advice about what works leans toward the winners.
Logs of an AI feature are dominated by conversations people finished. Those who gave up after a bad first answer leave short sessions that are easy to filter out as noise.