The Briefcase Technique: Making Your OSS Experiences Compound
No two OSS projects are the same. One might have a different vendor stack, another a completely different organisational structure, another a different operating model
Data integrity law #1 – Natural Decay – When being handled, the accuracy / integrity of a data set tends to degrade over time.
Data integrity law #2 – The Need for Curation – To prevent rule #1 from making the data unusable, the data needs to be curated.
Data integrity law #3 – The Cost of Curation – Curating data always carries a cost.
Data integrity law #4 – The Cost of Scale and Referential Complexity – The more data and the more referential integrity (ie cross-linking) the greater the costs.
Data integrity law #5 – Duplication-driven Decay – If the same data is maintained in more than one place (without automated synchronisation), the faster the decay time of law #1 and the higher the cost of law #3.
In other words, when the same fact is maintained in more than one place, integrity risk and curation cost increase unless authority, synchronisation and reconciliation rules are explicit
Data integrity law #6 – Data Minimisation (Minimum Viable Data – MVD) – To reduce costs and optimise integrity, retain only essential data, don’t duplicate it and keep cross-linking to a minimum.
The problem with law #6 is that it’s the cross-linking that often unearths the most dramatic insights.
Data integrity law #7- Isolation of Analytical Workloads – Dougie Stevenson rightly suggested a seventh data integrity rule – always use data snapshots rather than production databases to work on your data for BI purposes such as building new reports
Data integrity Law #8: Data integrity is relative to purpose – No dataset is universally high quality. It is only fit or unfit for a stated purpose, decision and tolerance. This is the distinction between correct data and decision-ready data.
Data integrity Law #9: Trust cannot exceed provenance – The trust placed in data cannot exceed the evidence of where it came from, when it was observed and if/how it was transformed
Data integrity Law #10: Errors become more expensive the further they propagate – The further a defect travels from its point of creation, the harder and more expensive it becomes to detect, explain and repair
Data integrity Law #11: Reconciliation and inference cannot manufacture truth – Where sources disagree, reconciliation requires an explicit authority, precedence rule or evidence-based decision. Matching alone cannot determine truth
Data integrity Law #12: Unrepresented uncertainty becomes false certainty – If uncertainty is not explicitly represented, downstream users and systems will eventually treat uncertain data as fact
Data integrity Law #13: Automation multiplies integrity – Automation doesn’t remove data-integrity problems. It multiplies their impact and speed
No two OSS projects are the same. One might have a different vendor stack, another a completely different organisational structure, another a different operating model
For decades, organisations have argued over whether IT or operations (OT) teams should control the OSS environment, as though it’s a binary decision. Giving one
Given the topical theme of the World Cup final, we’ll go with a soccer story today. For the World Cup final, do you think Argentina
For decades, scale gave large telcos purchasing power, infrastructure reach, extensive capability and millions of customers. It became one of the world’s most powerful and
2 Responses
Nice. I tend to agree.
When I do BI sorts of things, reporting, etc. I want to leave the reference data alone and use snapshots to my sort of work.
In the snapshots – I consider them to be just that – SNAPSHOTS.
Anyway, Application data structures may be be the right thing for Reporting… 😉
Great additional advice Dougie.
Especially in highly-available systems, like we tend to use, working with an offline snapshot of data is a great rule too!!