Take a look at the image below. If you were told that the image showed where planes had taken hits in WWII before returning to base and were asked to recommend locations on the plane where armour should be strengthened, where would you choose? Would you choose to strengthen behind the cockpit and on the wingtips?
“During World War II, the statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire. Researchers from the Center for Naval Analyses had conducted a study of the damage done to aircraft that had returned from missions, and had recommended that armor be added to the areas that showed the most damage. Wald noted that the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Wald proposed that the Navy instead reinforce the areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost.” Wikipedia.
Survivorship bias is prevalent in OSS too. We have complex processes (assurance, fulfilment and inventory) with many possible variants. Too many variants to test them all. Too many variants to catch them all. And if we don’t catch them, we can’t measure or process them – Survivorship bias.
Another example is in telco IVRs. Their designers are often asked to make traversing the decision tree so complicated that callers drop off the call before reaching an agent (that costs the telcos money to man the call centres). The sentiments of those dropping callers aren’t measured. Oftentimes, these callers are probably analogous to the downed planes of WWII (ie they’re the ones you actually want data on).
In the current age of omnichannel communications there are more cracks for customers to fall into than ever before because there is generally no single tool measuring journeys through all channels and watching all hand-offs between systems.
Do you think survivorship bias might be prevalent in your OSS?