Make no mistake: Big data is promising, exciting, and effective—when done right. Once considered an overhyped buzzword, it’s now a potential tool that leaders in every vertical want to harness. Unfortunately, the majority of new big data projects—about 55% of them, according to Gartner—are shuttered before they even get off the ground.
Generally, big data projects fail because they are missing at least one of three things: a holistic strategy; an agile data platform and infrastructure; or a collaborative, empowered team.
1-A Strategy for Navigating “Unknown Unknowns”
Many enterprises develop their big data platforms to solve one single, nebulous challenge, such as improving their supply chain’s efficiency or ensuring customer retention.
But big data is a major investment and should always be part of a holistic business strategy. A laser focus isn’t worth it—big data can tackle a host of problems, but successful implementation requires an open, experimental, and collaborative mindset. That’s because we’re on the frontier, moving from “known unknowns”—where we know what question we’re asking, and the only mystery is the answer—to “unknown unknowns,” where we don’t even know the question yet.
As an example, consider customer loyalty. These days, it’s no longer possible to know what sources you’ll require ahead of time to understand customer sentiments. Your customers interact with your business through innumerable touchpoints, and without big data, it’s nearly impossible to quantify the value of each one. To avoid this problem, begin your big data project with a willingness to experiment. Instead of putting all your eggs in one basket and hoping for a breakthrough innovation that earns $200 million, conduct 20 agile experiments that each result in a $20 million saving.
Over time, smaller wins can deliver more consistent gains than betting it all on one massive project.
This type of approach will also create a collaborative network effect, helping employees overcome perceived resistance to big data adoption from entrenched interests and legacy processes. Through experimentation, the enterprise will learn more while reducing its overall risk—a win-win.
2-An Agile Infrastructure
Big data is a long-term strategic investment, and it requires the right infrastructure to be successful. You simply can’t make a piecemeal investment in big data: In fact, that’s a mistake that’s undermined many projects.
Avoid that problem by deciding from the outset whether your interest in big data is significant enough to justify the time, cost, and labor involved. Then, approach the task of building an agile big data infrastructure with a minimum of coupling between the application and the infrastructure; this allows any component to be replaced or substituted with minimal effort, so the infrastructure can sustain itself for years to come.
3-A Collaborative, Empowered Team
One way to avoid investing in the soon-to-be-
obsolete is to ensure that the right people are in charge of design and implementation. That brings us to the final pitfall: You’ll likely need to hire or contract outside of your organization from the earliest stages, and should consider that a necessary cost of any big data project.
But even a skilled team won’t be enough unless they’re empowered to make decisions and guide overall strategy. Your big data team must have the authority to connect and work across siloed departments in order to drive meaningful business results. After all, if your data specialists can’t collaborate with actual decision makers, your investment will be a waste.
Businesses are scrambling to incorporate big data into their decision-making processes, often without fully considering the most critical aspects—including an experimental strategy, an agile infrastructure, and the empowered personnel who can actually engineer it. Although many big data projects don’t succeed, it’s clear that they fail for avoidable reasons. By making the right decisions now, you can build a big data infrastructure that remains at the cutting edge, leading to insights that will help your business remain competitive for years to come.