The business intelligence (BI) market is big: at least $10 billion in 2008 and much more if you include data warehousing projects. The tough economic environment may slow the growth of the BI market, but cost constraints, compliance and similar measures demanded by the current economy require accurate and timely business data, so BI is expected to remain a vigorous market segment regardless of the macro-economic situation.
Recent years have seen an incredible amount of consolidation as the big enterprise software players-most notably Oracle, IBM and SAP-made aggressive acquisition moves into the space. Oracle acquired Hyperion, IBM acquired Cognos and SAP acquired Business Objects. Microsoft also has been very active in the BI space with their analysis and reporting services offerings, and has acquired data warehousing appliance vendor DATAllegro.
Innovation in the business intelligence space is intense. The emergence of columnar databases, such as Vertica, potentially will shake up the currently RDBMS-dominated data warehousing world by providing databases engineered towards analytic efficiency rather than transaction processing. On the analysis side, companies such as QlikView are successfully promoting in-memory desktop analytic solutions, and Microsoft is working on similar features within its "Project Gemini."
With all this market activity, you might be forgiven for thinking that users are well-served by the wealth of business intelligence offerings. However, in reality, a lot of "real" business intelligence is still being performed at the departmental level using the decades old spreadsheet technology.
BI solutions built at the departmental or individual levels using Excel (perhaps with a dash of Microsoft Access) are sometimes called "spreadmarts." Almost all organizations have spreadmart solutions at some level, regardless of how much they have invested in enterprise business intelligence solutions.
Spreadmart solutions arise because of a fundamental incompatibility between the corporate requirements of flexibility and control. Enterprise BI solutions emphasize consistency and control: enforcing consistent policies, classifications, compliance and calculations. However, in the real business world, individual elements of the enterprise need to react quickly to changing circumstances-which may require changing the premises or methods of the standard analyses. Furthermore, enterprise BI typically falls under enterprise change control-evolving the corporate solution requires lead times that simply aren't always acceptable for the departmental BI consumer.
Spreadmarts typically extract data from corporate BI solutions or data warehouses, merge that data with information from departmental or other sources, and analyze the data using unique departmental categorization and calculation schemes. It is indisputable that top level management needs a consistent view of overall business operations. It is also true that other areas of the business need to be able to analyze data flexibly to address specific business objectives. Spreadmarts serve a valid need; however, they come at a cost. Spreadmarts from different areas of the same organization are unlikely to agree, while the creation and maintenance of the spreadsheet views is typically a manually-intensive operation that creates a significant, though often hidden, cost.
As long as the spreadsheet exists as a business tool, we can be confident that numerate users will create their own reporting spreadsheets. However, the gap between what business intelligence solutions can provide today and what the users of spreadmarts need clearly is too wide. We can hope and expect that BI vendors soon will come up with ways to provide the flexibility of the spreadmart with the consistency and control demanded by the enterprise.