In the early days of data warehousing, lines were very simple. The data warehouse reflected the business. In providing this reflection, data was summarized, data was cleansed, data was standardized, and data was even drastically reformatted for legibility and reporting usage. But, the big rule of thumb, the never-to-be-crossed line, was that the data warehouse did not create new data. Taking incoming flags and changing blanks and 1’s or 0’s and 1’s or blanks and A’s or whatever into a consistent “Y” or “N” was not creating data. Taking incoming code sets and expanding them to include an explicit value representing “unknown” was not creating data. Turning four- or six- digit numbers into a proper date data type was not creating new data. Creating a mailing list of customers and tracking response rates—that was creating new data. And, if data was created, then you were dealing with an operational solution outside of the data warehouse.
But that was then. Those lines of whether it “is-a-data-warehouse/is-not-a-data-warehouse” blurred with the rise of an alphabet soup of things such as ODS, CRM, MDM, ERP, etc. The scope of the data warehouse team’s responsibilities refocused on a broader area expressed as business intelligence, and included data scientists and others who created new numbers for describing new relationships, so that the old tasks for building a data warehouse started seeming a bit quaint. Albeit quaint, those tasks are still a vital need for most organizations. But in today’s world, the accountabilities covered by this particular solution team have expanded.
Extract, transform, and load (ETL) processing, in building data warehouses and/or data marts, is a critical component of every business intelligence arena. And some very specialized skills are used in building those components effectively. Along with setting up those data stores, reporting skills are also important, both in building those extra complex reports and/or in enabling a reporting tool to provide for user self-service reporting. However, even all of this ETL and report support is only a partial aspect in provisioning the business intelligence cupboard. Supporting business functions such as marketing, via mailing lists, customer retention programs, or any other of dozens of possibilities likely falls into the bailiwick of the business intelligence group. It would not be unusual for the business intelligence area to also end up holding the bag of responsibility for enabling and managing master data and/or local reference data. Even the administration of various aspects of overall data quality can be under the very broad and general umbrella that might be labelled business intelligence.
Managers need to be aware of these distinctions. Obviously, if a manager believes that certain functions belong in other areas, those managers can make those cases. However, it is not unusual for any group to have responsibilities that are quite far-reaching. Still, the distinctions between the various tasks are of critical importance. Differing skills need to be applied to each different kind of task. Report developers should not be tasked with creating screens for updating reference tables, and ETL specialists should not be tasked with writing web-based applications. Today, a much larger array of skills and specialties is necessary to keep a business intelligence team functioning within an organization. One could go overboard in the opposite direction and say that business intelligence is everyone’s job; but having a specialist perform the proper task will always provide a more optimal result than trying to make everyone a generalist doing any or every task at hand. And, a good business intelligence manager will work hard on applying the right tool to the right problem instead of being that silly person who only has a hammer.
Todd Schraml has more than 20 years of IT management, project development, business analysis, and database design experience across many industries from telecommunications to healthcare. He can be reached at TWSchraml@gmail.com.