Survivorship Bias in Data Analysis

Survivorship bias is drawing conclusions from only the entities that made it through some filter — surviving planes, active customers, successful companies — while the failures silently drop out of your dataset. The fix is always the same: find the missing denominator.

What Is the WWII Bomber Example?

In WWII, analysts mapped bullet holes on returning bombers and proposed armoring the most-hit areas; statistician Abraham Wald pointed out they should armor the areas with no holes, because planes hit there never made it back.

Data you have:  returning planes → holes on wings, fuselage
Data you need:  ALL planes       → where were the lost ones hit?

Holes on survivors mark where a plane CAN take damage
and still fly home. The clean spots (engines, cockpit)
mark the fatal zones — invisible in the surviving sample.

The dataset wasn't wrong, it was censored: the selection process (making it home) was correlated with the very thing being measured.

How Does Survivorship Bias Show Up in Churn and Customer Analytics?

Any analysis restricted to current customers is a survivor-only sample: the churned users who found onboarding confusing or pricing too high are exactly the ones missing from the data.

Classic traps:

"Our NPS is 62!"            → surveyed active users only;
                              churned detractors already left.
"Feature X users retain 3x" → engaged users adopt features AND
                              retain; the feature may cause neither.
"Avg customer tenure is 4y" → computed on current customers,
                              who are long-tenured by definition.
"Top firms all do Y"        → so did hundreds of dead firms
                              nobody profiled.

The right unit of analysis is the entry cohort — everyone who started, not everyone who remains. Track each cohort forward and measure what fraction survived and why.

How Do You Detect a Missing Denominator?

Ask "who had the chance to be in this dataset but isn't?" and check whether the row count matches the population that entered the process, not just the ones that finished it.

-- Survivor-only (biased): averages over remaining users
SELECT AVG(satisfaction) FROM users WHERE status = 'active';

-- Full cohort (honest): denominator = everyone who signed up
SELECT signup_month,
       COUNT(*)                                    AS started,
       COUNT(*) FILTER (WHERE status = 'active')   AS survived,
       COUNT(*) FILTER (WHERE status = 'active')::float
         / COUNT(*)                                AS survival_rate
FROM users
GROUP BY signup_month;

If a metric's denominator is "rows in the table" rather than "entities that entered the funnel," assume survivorship bias until proven otherwise. Backtests, app-store reviews, and case studies all fail this check by default.

Common Pitfalls

• Surveying only active users: satisfaction and NPS skew high because the unhappy have already churned. Survey churned users separately.
• Copying "what successful companies do": without the failure group, you can't tell practices that cause success from ones shared by winners and losers alike.
• Deleting churned users from the warehouse: hard deletes make the bias permanent. Soft-delete with a status flag and churn date.
• Fund or product performance averages: discontinued funds and killed products vanish from the index, inflating the average of what's left.