Think about the last time you filled out a paper form and contrast that with how many times you’ve filled out forms online. We live in an era where everything is digital. Forms are online, every click is captured, and even personal lives are documented on social media. The first wave of digitization led to more BI and better data-driven decisions. But, as we head into 2020, the focus has shifted from BI to operational analytics.
Traditional BI was focused on enabling executives to make decisions using historical data. It was accelerated by technologies such as Hadoop, which were built for scale but could not deliver results.
in real time. In contrast, operational analytics enables the operational teams on the ground to improve efficiencies every day. For example, UPS added real-time package tracking in the form of a
“Follow My Delivery” feature for users, but that same data is also incredibly useful for automatically routing their drivers more efficiently. Automation changes the very nature of work—both what we do and how we do things. When routine work gets automated, it frees up people’s time so they can instead focus on making better operational decisions. This creates a hunger for data and causes everyone to become an operational analyst.
Operational analytics systems support rapid decision-making and enable automation of actions based on real-time data. You collect new data from your data sources, and it all streams into your operational data engine. Your interactive applications query the same data engine to fetch insights from your dataset in real time, and you then use that intelligence to provide a better overall experience to your users.
A New Category of Data ?Processing Systems
Operational analytics gives rise to a set of requirements that, when taken together, is not fully addressed by existing transactional databases or warehouses. These requirements are the following:
- Support for complex queries such as joins, aggregations, sorting, and relevance
- Low data latency with new data showing up in seconds, not hours
- Low query latency with query responses in milliseconds
- Support for high-query volumeto address the needs of demanding operational teams
- Capabilities for live sync with data sources to always be the source of truth
- Support for mixed types in order to sync different types of data from new sources without the need for manual data preparation
This means there is a third category of data processing tools that businesses must think about in addition to their transactional databases and data warehouses. Transactional databases can support simple queries on small datasets, while data warehouses are good for long-running complex queries on large datasets. But data-driven automation requires fast, complex queries on real-time datasets. Not surprisingly, most of the recent innovation in the data space has been in streaming data, real-time processing, and operational intelligence (with software such as Apache Kafka, Spark, Elasticsearch, and Rockset).
Cloud Is the Enabler for ?Operational Analytics
The cloud enables new types of distributed data systems that can make it economically feasible to collect and process data in real time and at scale for every operational decision. In the cloud, the price of using 1 CPU for 100 minutes is the same as that of using 100 CPUs for 1 minute. If a data processing task that takes 100 minutes on a single CPU can be reconfigured to run in parallel on 100 CPUs in 1 minute, then the price of computing this task remains the same. However, suddenly businesses can get results in seconds instead of hours.
Cloud-native data platforms scale dynamically to make use of available cloud resources. This means a data request needs to be parallelized and the hardware required to run it must be instantly acquired. Once the necessary tasks are scheduled and the results returned, the platform promptly sheds the hardware resources used for that request. Simply processing in parallel does not make a system cloud-friendly. Hadoop was a parallel-processing system, but its focus was on optimizing throughput of data processed within a fixed set of pre-acquired resources. Similarly, many other pre-cloud systems were designed for a world in which the underlying hardware, on which they run, was fixed. The new breed of serverless search and analytics systems employs parallel algorithms at every opportunity until systems are hardware-resource bound. This means a lot of data can be crunched in near-real time. You don’t get these advantages by deploying traditional software onto cloud nodes; you need systems that are built for the cloud from the ground up.
Another reason operational analytics demands cloud-based systems is the fact that it has unpredictable patterns. When working with real-time data, the shape of the data itself is fluid. There tends to be flash flooding of incoming data and the access pattern also tends to vary depending on what problems the operational team is solving at any given point in time.
Opportunities in Every Part of the Business
Let us consider three specific examples of operational analytics in action in the product, marketing, and sales departments. It is easy to extrapolate and see how other functions such as customer support, supply chain, and finance can also use operational analytics to automate processes.
Product Management: In the past, product managers had to interview customers, survey the market, and often make intuitive product road map decisions. In the digital era, it is possible to track user actions, analyze clickstreams, and make product management decisions using data. Increasingly, customer success teams are closely tied in with product management as they collaborate on things such as chatbots that are automating customer interactions, gathering real-time feedback, and reducing friction in the customer journey. “Analytics and continuous intelligence fuel the constant evolution of products, and continuous DevOps delivers weekly or sometimes even daily product updates,” said Mark Raskino, research vice president and distinguished analyst at Gartner. “That’s why digital product management supersedes IT project management.”
Marketing: The traditional format of using focus groups to launch massive campaigns is being replaced by the use of growth hacking, where real-world experiments are run, data is analyzed in real time, and money is quickly diverted to the campaigns that show results. The bulk of the marketing budget is no longer spent on TV and radio ads and billboards. Instead, it is spent on Google, Facebook, and other digital platforms, where clicks and user behavior can be analyzed in real time. The rise of the “martech” stack is evidence that marketing teams are now using data to automate their way to more customers.
Sales: There was a time when enterprise customers were eager to learn more about new products from a salesperson. Today, they get their information online and see reviews from their peers first, then try the product themselves, and finally engage with a salesperson when the decision is almost made. The role of salespeople is to make sure that real-time data is used to track which customers are getting ready for that purchase decision, understand the customer’s context, and engage at exactly the right time, in exactly the right way, to close the deal.
A leading venture capital firm, Andreesen Horowitz, has predicted that operational analytics will transform how information workers in the enterprise operate. Having access to operational analytics changes the game for every functional role. The key is to not just collect real-time data but to also make sure that it is processed in real time and is easily consumable both by the software that drives automation and the people who make decisions every day. When teams are empowered with operational intelligence, the business as a whole can move toward a level of efficiency that was simply impossible in an earlier era.