![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/graham-zabel-cropped-colour-tt-width-300-height-327-fill-0-crop-0-bgcolor-eeeeee-except_gif-1.jpg\")
<\/figure>\n\n\n\n![\"Sustainable](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/A-Cloud-Linkedin-Haiilo-47.png\")
<\/a><\/figure>\n\n\t\t\n\t\tThere may also be tool migrations or upgrades in progress, further complicating the spread of bad data. \u201cWe’re migrating from Golden Source 1 to Golden Source 2, so we are not allowing any more changes to Golden Source 1. We\u2019ll clean it up later.\u201d Effectively this says that we will live with the virus, and the spread of bad data throughout our systems.<\/p>\n\n\n\n
Also, there is the Fear Of Deleting Data (FODD)<\/strong>, making it even more difficult to \u201ckill\u201d the virus. (Follow us to read our upcoming Anti-Pattern #3 for more details.)<\/p>\n\n\n\n While the management of old, stale or redundant data may be a low priority issue, due to the Bad Data Virus, the problem may be a magnitude larger than you realise in terms of:<\/p>\n\n\n\n The problem of storing bad data may seem minor at team scale, but is serious at enterprise scale, and very serious at national and global scale.<\/p>\n\n\n\n To avoid spreading the Bad Data Virus, bad data needs to be cleaned where it originates – in the Golden Source system. Any cleaning of the bad data in downstream systems is potentially wasted activity, as the Golden Source will still have the bad data so it will reappear in downstream systems after the next feed or data transfer.<\/p>\n\n\n\n Due to the Fear of Deleting Data (FODD)<\/strong>, it is highly unlikely and possibly dangerous for data to be maintained and cleaned manually. Data maintenance policies must be automated.<\/strong><\/p>\n\n\n\n Data Ingress<\/strong><\/p>\n\n\n\n In order to stop bad data getting into our systems, we need to design automated guardrails for data entry.<\/p>\n\n\n\n For example, all data should have an owner. Data should not be allowed into the system without an assigned owner. The owner should be a product, service or team. The owner should NOT be an individual. Individuals come and go, but the product and\/or service should be long-lived. If data has an owner, it is easier to ask the owner what to do with the data. If data does not have an owner, it is very difficult to determine the importance or usefulness of the data, so the data will likely remain in our systems due to FODD.<\/p>\n\n\n\n Data Storage<\/strong><\/p>\n\n\n\n In order to keep our data stores clean, we need to define automated clean up policies based on well publicised data retention policies. Automated data maintenance policies provide comfort to the teams and services because the policies have been approved and are well understood. This eliminates FODD.<\/p>\n\n\n\n For example, we may decide that \u201cSnapshot\u201d (or non-persistent) binary artifacts from our continuous build systems only need to reside in our Artifact Repository systems for 3 months before they can be deleted. Once this policy is in place, snapshot artifacts are automatically cleaned up and developers know that these artifacts won\u2019t last long. If they need persistent, long-lasting artifacts they should flag the artifacts as \u201cRelease\u201d artifacts. When these policies are in place and well understood, delivery teams will embed their own processes to ensure important artifacts are persisted properly.<\/p>\n\n\n\n![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/Bad-Data-Virus.png\")
<\/figure>\n\n\n\n\n
Solution<\/h2>\n\n\n\n
Automated Data Maintenance<\/h3>\n\n\n\n