Why a Majority of Data Warehouse Projects Fail—and What Businesses Can Do

Today’s businesses understand the power of data. It surrounds every aspect of their operations from marketing and sales to new product design, and even the onboarding of new employees. Captur­ing, analyzing, and utilizing the available data at every possible opportunity is what defines modern business success.

As companies embrace digital trans­formation across the enterprise, it is data and the effective use of it that determines whether new technologies such as AI, auto­mation, and analytics will be successful. The importance of having an effective data man­agement and data warehousing operation is more critical than ever before.

That said, setting up and implement­ing a proper data management and ware­housing system has proven to be elusive for many businesses. There are several key obstacles and recurring challenges that businesses face and struggle to overcome. Siloed applications and data, outdated leg­acy technologies, and manual (and slow) data movement have been found to block the sought-after benefits and often cost millions to fix or work around.

Taking a Deeper Look

In a recent study, independent market research firm Vanson Bourne surveyed hun­dreds of businesses across the U.S. and U.K., and confirmed that businesses are still strug­gling to get a handle on the best way to man­age their evolving data challenges. Almost all respondents (93%) believed that improve­ments are needed in how they collect, man­age, store, and analyze data.

The group of IT decision makers sur­veyed in the report were very aware of the problems that lay ahead of them, with 83% saying that they weren’t fully satisfied with their data warehousing initiatives. In fact, the businesses involved in the research reported that 42% of their existing data manage­ment and warehousing processes that likely could be automated, were currently being done manually, consuming valuable time, resources, and money.

Nearly nine out of 10 (88%) said they were experiencing challenges trying to load data into data warehouses, with the biggest inhibitors being legacy technol­ogy (49%), complex data types and for­mats (44%), data silos (40%), and data access issues tied to regulatory require­ments (40%).

They went on to say that nearly half (48%) of the data migrated into their businesses’ data warehouses (or other storage solutions) required cleaning before it could be useful. The findings showed that this process was a large contributor to more than four hours of time lost per employee each week due to the need to work on issues concerning miss­ing data, duplicate data, or data that needed reformatting.

Four Obstacles to Data Success

According to the study, there are four key obstacles recurring in most busi­nesses which are stalling data warehous­ing progress and success. These are dis­connected data silos, slow loading of the data warehouse, time-consuming data preparation processes, and a need for more automation of their core data man­agement activities.

Disconnected Data Silos

As businesses look to accelerate their digital transformation initiatives, cloud-based products and solutions clearly offer the shortest path to success and a return on investment. However, this quick adop­tion of cloud tools, mixed with the democ­ratization of technology use throughout a business (where department leads are becoming more likely to be responsible for purchasing or developing technologies for their own department’s use), often creates an unintended problem: application sprawl and data silos.

There is a proliferation of data silos within today’s businesses as each line of business independently pursues “agility.” Each of these silos represents another distinct area from which data must be pulled, integrated, and analyzed in order to be used correctly to empower decision making. Silos led by departmental leads often don’t adhere to a common set of businesswide standards, making the task of connecting all of the data even more potentially challenging. In addition, because of the democratization effect of cloud technologies, there may be valu­able data silos that IT doesn’t even know about, hiding out in their network.

The research found that the average business has 115 distinct applications and data sources across their organization— and almost half of them are disconnected from one another. What’s more, 89% of those surveyed in the report worried that these silos are holding them back from dig­ital modernization success.

Slow Loading of the Data Warehouse

Once discovered, data needs to be col­lected, reviewed, and analyzed across all departments. But discovering and using the data hidden in silos is only one of the prob­lems businesses face. The majority of busi­nesses still utilize legacy technologies in some way, shape, or form in their network, and about half indicated that their legacy tech­nology is indeed restricting data movement. Legacy technologies—especially those that have different histories, formats, and ways of operating—can create logistical nightmares.

If the technologies will not speak to each other, or to the tools being used to collect the data, then extra time-consuming work needs to be done to get said data where it needs to be in the data warehouse. The research found that some 88% of respondents are struggling with effectively loading data into their data warehouses, which is hindering data access and delaying realizing data-driven insights.

In addition, due to the time-consum­ing and resource-draining work that needs to be done, the likelihood of the data ulti­mately being ready for analytics when needed recedes.

More Automation of Core Data Management Activities

Many businesses face additional work after locating their data and migrating it into their data warehouse. The research found that 48% of the data that is migrated into their data management and warehousing solutions requires cleaning before it can be useful.

The data also indicates that businesses are currently using an average of four different tools to integrate and move data in and out of data storage and warehouse solutions. It’s easy to see how using multiple, different data integration and movement tools introduces risk, duplication, and error to the process, while also intensifying the labor needed to get a return on investment.

As long as these issues persist, the data that is held in data warehouses is unlikely to be of good enough quality to deliver robust analysis and a business’ decision making will be flawed.

More Automation of Core Data Management Activities

To understand how important auto­mation is to improving the allocation and optimization of a business’ data, a closer look needs to be taken at the statistic that 42% believe that data management pro­cesses that could easily be automated are still being done manually.

For every process that is being done manually but could be automated, valu­able time, resources, and money are being squandered—and business opportunities are lost in the process.

Automating manual, time-consum­ing data management processes, such as the integration of disparate applications and data sources, or the movement of quality data into the data warehouse, saves time and money, while reducing the time it takes to see value from the process. Automation can accelerate all stages of data management and data warehousing, including data collection, integration, preparation, storage, sharing, and analysis.

It can even speed up the identification of siloed data sources and the migration of that data from legacy systems to the data warehouse. Duplicate, erroneous, and missing data can be identified quickly, giv­ing a business details on what should be done to make repairs.

How to Make a Data Warehouse Project Work

Put simply, in order to get data ware­housing right, businesses must break down data silos, retire legacy tools, auto­mate manual processes, and accelerate the integration and movement of data across the enterprise. This is easier said than done, but the research indicates there is a commitment to making this happen, with 76% saying that data budgets have increased since last year, despite or per­haps in response to the current pandemic.

With this in mind, there are seven key steps businesses should follow to ensure data warehousing success:

  1. Establish a Data Culture—Does every­one in the business understand the power of data? Start by investing in talent, training, and support to drive a strong data culture.
  2. Define Objecdtives—What are the data goals? Is the business looking to improve the customer experience? Or drive operational efficiencies? Be clear on desired outcomes before starting the process, choos­ing technologies, or selecting teams.
  3. Review Existing Technology—What systems does the business already have in place? What’s working, what isn’t, and what’s missing? There may very well be solutions in place that can be leveraged without having to acquire new, expensive solutions.
  4. Define Data Sources Audit—applica­tions, the network, and any possible data sources throughout the business. What is in use most frequently, and what hasn’t been touched?
  5. Integrate and Automate Choose—an integration platform that can handle both applications and data, embraces AI and self-service, and automates as much as possi­ble. It is important to consider the ability for the platform to scale and grow as the busi­ness does.
  6. Report and Analyze Use—visual, easy-to-understand charts, graphs, maps, and dashboards to ensure different busi­ness teams have access to the results of the data project. It is important they can use it to understand trends and make informed decisions. Seeing the business value in data will help ensure they continue to support its collection and use.
  7. Monitor Continuous—improvement is key to maintaining the success of a data warehouse. Monitor, measure, and iterate to ensure systems and processes keep run­ning optimally.

Only when these technologies and pro­cesses are in place, can teams truly harness the full power of their data to drive bet­ter decisions, actions, and outcomes. The future success of the business depends on getting this right. n