There was a time when what you saw was what you got. Things were only skin-deep, simple. Building up the components of a business intelligence area was very straight-forward. A staging area was a staging area; an operational data store was an operational data store. But like buying a pitcher of beer for $2, or gas for less than a dollar per gallon, those days are gone.
The dynamics have changed, things are more federated, IT must accept more than one standard tool. A suite of varying tools and platforms, each bringing slightly different strengths, and weaknesses, has become the norm. It may have started on the operational side, particularly web development, where for a while new languages and toolsets appeared overnight, and were grabbed and used often indiscriminately.
This multiplicity is true of business intelligence data analysis tools, as well as in how we architect the data warehousing ecosystems. A data warehouse architect should not be surprised to be told to find ways to coherently allow for some data to be persisted in the cloud, or some cloud variant, while other data remains in the traditional in-house DBMS. Staging areas are more and more often trialed, or permanently migrated, over to Hadoop environments to leverage Hadoop’s particular strengths, or to avoid using up expensive RDBMS resources. The complexity arises as these new tools and approaches are not replacing others in a wholesale fashion, but instead becoming additional elements expanding the environment. An operational data store component may need to be considered an umbrella for multiple databases or even database platforms, that as a group are fulfilling the organization’s ODS function.
Allowing for multiple tools, is not to say “anything goes” as that unfettered approach would lead to chaos, which hopefully is what a good architect works very hard to prevent. Being flexible is not the same thing as being indifferent. Allowing for a multiplicity in tools and approaches means each variant is there for a known and agreed upon purpose. That flexibility means t such purposes are both documented and enforced. For example, it is not okay for an ODS to start being used as a data mart, or a sandbox to become the data warehouse. But, it would be okay for new ODS elements to be placed in any of the organization supported DBMS or other stores, for appropriate reasons. Those rationalizations could include cloud, Hadoop, or other options depending on the agreed upon landscape.
Some IT shops try to battle for conformity and standardization, claiming one tool is the only one approved, while refusing to support users when they color outside the lines. In an efficiency-driven approach, fewer tools means fewer skills that the IT area must support, which we are told means a lower cost to the organization. But approved or not, the business users bring in shiny new tools even when they are “unapproved." These users have their reasons, and their needs. IT shop dictates cannot make these needs go away. A more forward-thinking perspective embraces multiple tools, not an open door to any and all tools, but allowing a limited set of tools, each focused on their own niche. Failure to accept varying groupings of tools and configurations is only highlighting antiquated and detrimental policies. Users are not the only ones asking for change. Sometimes IT leadership may be driving for newer, less expensive options, hoping for a wholesale replacement and lower costs. Regardless of the driving forces, a bricolage is what we must now compose for an organization to survive and thrive. And if the chaos is properly navigated things can become both flexible and orderly.