Governing and managing big data are not easy tasks. In fact, let’s be honest—data management and governance for data of any size is no walk in the park. But, big data makes it even tougher. From integrating the disparate data sources of seemingly unending variety to curating the chaos in the heaps of unstructured data, managing the craziness we lovingly refer to as big data is not for the faint of heart. Even for those tough as nails, the challenges of big data management can be more than frustrating. So, now that you know you are not alone in dealing with this insanity, here are a few ways to make the frustrations of big data a little less intense.
1. Begin With the End in Mind
Requirements for big data projects should be based on clearly defined objectives and specific outcomes. Design and build with intended business use cases in mind. Include the folks that will be using big data to develop analytic models, build dashboards, derive insights, or drive business decisions in the projects from start to finish—after all, they will be the ones who will be using it. This will not only ensure that the design meets their needs and requirements, but it will also establish buy-in from the onset.
2. Use the Right Tool for the Right Job
Contrary to popular belief, Hadoop is not the only big data tool out there. While it is a very valuable tool, it has its limitations. Big data is going to be much easier to use across the company if there are many tools and options at the ready. There is a time and a place for cloud technologies, vendor-specific solutions, and open source. Don’t be afraid to use a healthy combination of tools and resources. And this references more than just technology too. Consider your options for outsourcing or contracting skills and talent when necessary. Not everything has to be an inside job—especially early on.
3. Visualize
Big data is overwhelming. Regardless of speed, size, or variation, when looking at individual data elements, it is just too much for the human brain to consume and understand. Invest in visualization technology that will change how you see your data. It will give you (and everyone else) the opportunity to look at your data in new ways. From recognizing new patterns to seeing data with greater meaning, visualization technology will drive the curious to ask new questions of your data and move your business forward, faster. Just remember—the more that is “under the hood” in the visualization software from an analytic and mathematical perspective, the more value you will gain for your business.
4. Allow Room for Experimentation
Data exploration is a great way to uncover new business possibilities and ask new questions of old data. Innovation requires failure as much as success, if not more. Allow employees the opportunity to try and try again. But, exploration and experimentation should happen outside of the operationalized environment. Experimentation can be messy. Just because it works in the “lab” doesn’t mean it will work at scale. Let your mad scientists know that there is a process for moving from the sandbox to the full-scale production environment (and stick to it!).
5. Cross-Pollinate
Big data projects are by no means one-man shows. In fact, some big data efforts are that of small armies. In order for big data projects to work in businesses, they require efforts across business units and contributions from business and IT alike. And, it is no longer a discussion of aligning business and IT. To achieve big data success, business, IT, and everyone in between will have to work together on the same objectives and goals—not aligned objectives. It is a subtle but very powerful difference. And as your big data projects change, so will your big data teams. Use the skills and talents throughout your entire company. If you want big data to work at the enterprise level, then make sure you include people from across the entire enterprise, as well.
Governing and managing big data will never be trouble-free or effortless. However, applying these best practices, even just a couple of them, will ease the burdens significantly. And as big data management starts to feel a little easier, the rewards will become greater.