Yesterday’s post discussed two waves of decisions stemming from our increasing obsession with data collection.
“…the first wave had [arisen] because we’d almost all prefer to make data-driven decisions (ie decisions based on “proof”) rather than “gut-feel” decisions.
We’re increasingly seeing a second wave come through – to use data not just to identify trends and guide our decisions, but to drive automated actions.”
Unfortunately, the second wave has an even greater need for data correctness / quality than we’ve experienced before.
The first wave allowed for human intervention after the collection of data. That meant human logic could be applied to any unexpected anomalies that appeared.
With the second wave, we don’t have that luxury. It’s all processed by the automation. Even learning algorithms struggle with “dirty data.” Therefore, the data needs to be perfect and the automation’s algorithm needs to flawlessly cope with all expected and unexpected data sets.
Our OSS have always had a dependence on data quality so we’ve responded with sophisticated ways of reconciling and maintaining data. But the human logic buffer afforded a “less than perfect” starting point, as long as we sought to get ever-closer to the “perfection” asymptote.
Does wave 2 require us to solve the problem from a fundamentally different starting point? We have to assume perfection akin to a checksum of correctness.
Perfection isn’t something I’m very qualified at, so I’m open to hearing your ideas. 😉