Unusual dynamics facilitate data cleansing that moves content from the source application into the data warehouse. Circumstances can arise that result in data considered either “wrong” or at least “not-as-desired” when viewed within the data warehouse, yet further investigation verifies that same data corresponds exactly with the source values. Meanwhile, some executive decision will decree that the data warehouse and dependent data stores must have the “right” data. By design, the data warehouse will not accurately reflect the source, or system-of-record. Obviously, one hopes these occasions occur only rarely. However, with a certainty that matches the sun setting at evening tide, all the circumstances, decisions, and reasoned thought which purposefully drove this data inconsistency will end up eventually forgotten. Time passes, an ambitious user checks the data warehouse against a source, system auditing stumbles into this specific set of data, or reprocessing of this source data (performed for any reason) creates “new” results that do not match the “old” results. These events can trigger considerable debate and endless discussions. Ideally, everyone involved agrees to and understands the previous choices; but in the final outcome, the same ground, the same options, and the same choices recycle themselves and waste peoples’ time.

Metadata expressly aids in dealing with “inconsistencies” such as those mentioned above. It may come as a surprise to some, but a metadata repository consists not only of data definitions, it also serves as a place to publish data-related business decisions - such as acknowledged cases where specific requests have demanded that retained data not match the source system. Certainly, when questions arise regarding why changes occurred, inquiries may be directed at someone who remembers the event, or those potentially knowledgeable individuals may be on vacation, on another project, or at another organization and therefore unavailable. Regardless of the circumstances, even assuming actual availability to field questions of these in-the-know folk, an “oral tradition” does not constitute the best approach for publishing such business data knowledge.

Having some form of data knowledge repository allows an organization the possibility for creating a shared memory of the current status of things and how things work, while also identifying significant decisions made to alter these things from normal expectations. Rich price tags on formal repository products normally frighten most organizations; but a virtual repository can exist, even if it just consists of a folder with a few documents and spreadsheets. Without at least some investment, minimally by someone who gathers and maintains the information, then nothing really exists, even virtually.

The important part worth remembering, whether dealing with a data repository as virtual or real, complex or simple, sparse or filled with enticing factoids, is that ultimately no data repository makes any impact within an organization without people accessing and using the information within the repository. Therefore, besides providing a place for metadata to exist, the people who need that knowledge must actively go browsing and finding what they need at any given time. To achieve this preferred state requires good salesmanship; users must accept looking for answers in the offered repository. Organizing things in a manner that aids these knowledge users in their efforts will more likely please those same knowledge seekers with how it all works, so they come back for more input the next time they have a question. Ideally, one hopes to develop a Pavlovian response, so that when a question comes up, a search gets done, producing a result that provides the reward, and thereby bringing value to the supplied structure and coordination of a metadata effort.

A good custodian of organizational data realizes the importance of considering the future when making today’s decisions. While an intention to have the data warehouse change values and yet not change the source system may seem reasonable and perhaps even obvious today, discerning who eventually might need to remember such decisions months or years down the road requires careful assessment, as well as determining by what means these presumed "rememberers” can expect to know the right answer when required. Good data custodians attempt to address those issues while also providing value in the present moment.

About the Author:

Todd Schraml is senior data architect and manager of ETL at Innovative Health Strategies, Inc. He can be reached at tschraml@ihsiq.com.