Customer and end-user expectations for interacting with computerized systems have evolved as data volume and higher transaction velocities are becoming commonplace for modern applications.
Organizations are being challenged as they attempt to harness all of this data—regardless of its source or size—and to garner actionable insight from it. This is known as data analytics.
Advanced analytical capabilities can be used to drive a wide range of applications, from operational applications such as fraud detection to strategic analysis such as predicting patient outcomes. Regardless of the applications, advanced analytics provides intelligence in the form of predictions, descriptions, scores, and profiles that help organizations better understand behaviors and trends.
Furthermore, the desire to move up the time-to-value for analytics projects is causing a move to more real-time event processing. By analyzing reams of data and uncovering patterns, intelligent algorithms can make reasonably solid predictions about what will occur in the future. This requires being adept enough to uncover patterns before changes occur. Of course, this does not always have to happen in real time.
Challenges for Advanced Analytics Projects
It is common to encounter problems along the way when implementing an analytics project. One of the first issues is likely to involve ensuring that organizational leaders embrace the ability to make decisions based on data instead of gut feelings. This means moving away from the HiPPO decision-making process, where the HiPPO is the Highest Paid Person in the Office at the time the decision needs to be made!
Things change so fast these days that it is impossible for humans to keep up with all of the changes. Cognitive computing applications that rely on analytics can ingest and understand vast amounts of data and keep up with the myriad of changes occurring daily, hourly, or even quicker. Armed with advice that is based on a thorough analysis of up-to-date data, executives can make informed decisions instead of what amounts to the “guesses” they are accustomed to making.
Nevertheless, most managers are used to making decisions based on their experience and intuition without necessarily having all of the facts. When analytics-based decision making is deployed, management can feel less involved and might balk. Without buy-in at an executive level, analytics projects can be very costly without delivering an ROI, because the output (which would deliver the ROI) is ignored.
Another potential difficulty involves managing and utilizing massive quantities of data. New data is being continuously created internally and many times, additional external data is purchased to augment existing business data. This explosion in data volume is one of the driving forces behind analytics. The more data that can be processed and analyzed, the better the analytics can be at finding useful patterns and predicting future behavior.
Data can increase in depth (more customers, transactions, etc.), and data can grow in width (where subject areas are added to enhance the analytic model). As the amount of data expands, the analytical modeling process can elongate. Clearly, performance can be an issue.
However, as data complexity and volumes grow, so does the cost of building analytic models. Before real modeling can happen, organizations with large data volumes face the challenge of getting their data into a form from which they can extract real business information. This data preparation is a very time-consuming step in the analytic development. Data gets extracted, subsets of data are created, these subsets are joined together, merged, aggregated, and transformed.
Real-time analytics can pose another interesting challenge because “real-time” requires immediate or nearly-immediate responsiveness. Market forces, customer requirements, governmental regulations, and technology changes collectively conspire to ensure that data that is not up-to-date is not acceptable. As a result, many organizations are constantly working to achieve real-time data access and analysis. As good as real-time analytics sounds, it comes with significant implementation challenges. One such challenge is reducing the latency between data creation and when it is recognized by analytics processes.
Time-to-market issues can be another potential pitfall of an advanced analytics project. A large part of any analytical process is the work involved with gathering, cleansing, and manipulating data required as input to the final model or analysis. This takes time, but this up-front work is essential to the overall success of any advanced analytics project.
From a technology perspective, managing the boatload of data and the performance of operations against that data can be an issue. Larger organizations typically rely on a mainframe computing environment to process their workload. But even in these cases the mainframe is not the only computing platform in use. And the desire to offload analytics to other platforms is often strong. However, for most mainframe users, most of the data resides on the mainframe. If analytics is performed on another platform moving large amounts of data to and from the mainframe can become a bottleneck. Good practices, and good software will be needed to ensure that efficient and effective data movement is in place.
But before investing in a lot of data movement off of the mainframe, consider evaluating the cost of keeping the data where it is and moving the processes to it (the analytics) versus the cost of moving the data to the process. Usually, the former will be more cost effective.
Taking advantage of more in-memory processes can also be an effective approach for managing analytical tasks. Technologies like Spark, which make greater use of memory to store and process data, are gaining in popularity. Of course, there are other in-memory technologies worth pursuing as well.
The Bottom Line
There are many new and intriguing possibilities for analytics that require an investment in learning and new technology. But the return on the investment can be sizable in terms of gaining insight into your business, and in better servicing your customers.