The honeymoon between business and big data is over. The end was conclusively noted when Gartner placed big data in its trough of disillusionment. We’ve reached a point where companies must figure out how to use big data analytics in profitable ways.
That is easier said than done. Given the enormous volume of data being generated these days, “signal” must be separated from a whole lot of “noise.” Across a wide variety of data types, extracting more value can require adopting a diverse set of analytic techniques. And the velocity with which big data accumulates and ages means analytic insights can improve business performance only to the extent they can be quickly brought into operations to drive actions.
So how can businesses use big data analytics to achieve better, faster decisions? Here’s a checklist of six best practices for obtaining more business value from the ever-growing volume, variety, and velocity of available data.
6 Best Practices for Getting Value from Big Data
1. Start with a business problem in mind.
Exploring huge amounts of data with Hadoop and other advanced analytic tools can be fun for your analytics team, but it can also be a waste of time and resources if the results do not translate into something that solves real-world business problems.
Identify projects that are both promising and practical, and take time to understand the various types of problems big data analytics can solve for your organization. For instance, there’s a lot of conversation around analyzing unstructured data such as video and speech. However, the most important source of big data for many businesses is consumer transactions, which tend to yield structured data. Payment card and loyalty program transactions, for example, produce abundant, timely streams of data. These are replete with granular details on the what, when, how much, and how often details of individual spending. And, keep in mind that the cost and complexity of analyzing huge amounts of structured data are often much lower than the cost and complexity of analyzing unstructured data.
The bottom line is that you need to find out what kind of business problem or challenge can be addressed with the data you have, and ensure that the data being analyzed is current, accurate, and offers real insight.
2. Look ahead to how you’ll deploy insights in different business functions.
To achieve real business value, you have to be able to operationalize the results of your analysis. Although this seems obvious, far too many projects are left gathering dust or subject to delays because it is too hard to leverage findings where they could provide value. The opportunity cost to a business—from all the suboptimal decisions made in the interim—can be immense.
Wise selection of data is critical. What looks wonderful in the lab may not be available or may be too expensive to obtain at the time needed for use in day-to-day business operations. Industry regulations too may affect where and how data can be used.
Analytic development teams must carefully consider how their models will be published and used by marketing, customer service, product development, or operations teams. Models that rely on manually intensive data processing steps, for instance, can cause problems at implementation. Such issues can have far-reaching effects, especially in regulated areas such as lending and insurance underwriting, where they make it difficult to explain and defend data-driven decisions to auditors and customers.
Technology advances are helping organizations to avoid these and other problems and to speed up analytic processes. Streamlined analytical methods not only reduce time to operational value but also make analytic results easier to share and reuse for multiple purposes.
In addition to being able to deploy analytics more quickly, having a mechanism in place to enforce best practices in model management helps to avoid development and implementation delays.
3. Leverage analytic innovation.
Innovations in big data processing and analytics are transforming how businesses get value from their customer data. We’re seeing a shift from approaches that supply periodic snapshots in the form of descriptive reports and dashboards (what happened) to systems that continuously analyze incoming data to produce predictions (what is likely to happen) and prescriptions (what to do about it) that are actionable in real time.
Many types of analytics will increasingly operate inside production streams. Relying less on persistent historical data, they will instead respond to changes in the current environment. Analytic outputs will be combined with complex event processing to enable very rapid responses to customer behavior.
Big data tools and infrastructure are also making it easier to apply machine learning techniques to explore huge datasets that include a wide variety of structured and unstructured data. The right balance of these techniques with human analytic and domain expertise not only lifts business performance but also improves the ability of companies to learn at a fast pace from data-driven experiments.
4. Embrace analytic diversity.
R, Python, Hive, Groovy, Scala, MATLAB, SQL, and SAS: One of the side effects of the exploding world of analytic innovation is that taking advantage of the latest techniques often requires learning a new set of tools. Analytic teams will inevitably need to use multiple development methods to deliver the insights the business needs. It’s also clear that combining different types of analytic techniques can deliver superior results.
To get multiple types of analytic models to work together in an efficient development environment and robust production environment, you need a flexible infrastructure that embraces diversity. Fundamental requirements include the ability to operationalize models authored by a wide range of tools by supporting extensible libraries, web services, and standards such as the Predictive Modeling Markup Language (PMML) and even the new Decision Model Notation standard. Centralized lifecycle management should extend across models, business rules, and analytic assets from any source.
At the risk of sounding hypocritical, it’s also very important to build a culture of documentation and control while leveraging all of these tools. Production use of analytics requires discipline and control while finding ways to not stifle creativity. That’s the balance successful organizations manage to find.
5. Leverage cloud services and productivity platforms.
Creating big data analytics no longer requires making a huge investment in expensive infrastructure and specialized skills. By leveraging cloud services, companies can let a dedicated third party securely handle the underlying systems and services, paying just for the capacity and services they need. Using open, hub-based architecture is a quicker, less costly way to improve cross-functional visibility and coordination compared with the traditional one-to-one systems integration.
Unless analytics capabilities are to interact only with applications, you will also need tools for packaging analytic services for business users. Today’s application development productivity platforms (available for on-site installation or via cloud services) provide everything needed to create complete applications, including user forms and workflows powered by the analytic models.
6. Balance automation with expertise.
More analytics doesn’t mean less need for human expertise. Analytic expertise informed by deep domain knowledge is essential for building effective predictive and decision-making models. Today’s shortage of analytic talent puts more pressure on organizations to ensure they engage with well-trained data scientists, either their own in-house experts or vendors with whom they choose to collaborate.
Make sure the people or businesses with whom you partner for your big data projects really understand the data that drives both the decisions and the building of the analytic models. With the expanding space of open source and commercial data science tools, newer “data scientists” often use these tools without a true understanding of how they work, what the parameters mean, and the impact they can have on your business decisions.
When analytics efforts deliver disappointing results, keep in mind that it is often because there is not enough, or the right, analytic expertise at hand. For more on why this kind of expertise is vital, I recommend two blog posts from my colleagues at FICO: Gerald Fahner on “Four Ways We Can Build Trust in Analytics” and Scott Zoldi on “Three Challenges in the Analytic Tech Crunch.”
Getting the Most From Big Data Analytics
The value of big data to business is easy to understand. But it’s not as easy to extract insights from immense stores and incoming streams of data in an actionable form—and in time to make a difference to your business. Fortunately, a set of reliable best practices can help you take advantage of the big data momentum.
This article first appeared in the Summer issue of Big Data Quarterly Magazine.
Image courtesy of Shutterstock.