ISSUES AND OBSTACLES
Data experts suggest the following ways to lay the groundwork for real-time analytics:
Data readiness. A challenge to this new level of analytics is getting the data ready. “The data itself often isn’t ready, as it’s fragmented, unclean, and siloed,” said Clayton. “Customers have attempted to implement customer data platforms to make this happen, but due to their focus on marketer needs for activation, data has become a second-class citizen.”
Data needs to be of the highest quality, timeliness, and relevance. “With many organizations still lacking the core data foundations they need to manage structured and unstructured data together, teams will still be missing key ingredients they need to fully trust the data they’re seeing—even in real time—to make the best decisions possible,” Prakash added.
Data freshness. Maintaining data freshness is another challenge. “Real-time delivery only provides value to the business when the data is complete, accurate, and ready at the time of use,” said Srinivasan.
Costs. “Scaling infrastructure to process millions of events per second while keeping costs manageable adds another layer of complexity, and many teams underestimate what it takes to achieve that level of performance,” said Srinivasan. In addition, integrating legacy systems can be costly, “because these systems may require custom solutions or expensive adapters or middleware, creating a complex architecture,” said Prakash.
Legacy integration. While there are several challenges that can slow down progress with developing and deploying real-time analytics, topping the list is the prevalence of legacy systems that were originally designed for batch workloads. “Tooling gaps largely show up when legacy systems need to be integrated,” said Prakash.
Data often “lives in silos across ERP, CRM, and custom apps because companies have adopted technology as it comes out without auditing what they have, need, or what the hot new tech will do in their day-to-day operations,” said Pytel.
Legacy systems also hamper real-time analytics integration into operational technology (OT). “Legacy OT systems are often fragmented and difficult to connect,” said Streit. “This has been exacerbated in recent years by the rapid deployment of new applications, control systems, and edge devices, all of which generate data in different formats and protocols.”
This variety “makes comprehensive, contextualized analytics more and more challenging,” Streit continued. “So, while industry certainly understands the value of real-time insights, it can be difficult, especially as organizations’ technology stacks and data silos continue to grow.”
People. Beyond technology, people and processes also need to be adapted to support real-time analytics. “Decision-making processes may be such that providing real-time analytics may not result in real-time decisions,” Prakash said.
RECOMMENDATIONS
Data experts suggest the following tips to pave the way for real-time analytics:
Get your data house in order. “The best thing companies can do for real time insights and analytics is to get their data house in order from the ground up, bring AI and GenAI into workflows, and make it so people can get trusted insights where they work,” said Prakash. “Don’t make people go to the dashboard—bring the data to them.”
Invest in “an upstream data management layer that continuously cleanses, standardizes, and unifies data across all sources,” said Clayton. “The goal is to make it so the data that is available can be trusted from the start, not require extra processing to make it useful for specific systems. Reverse ETL is not a stand-in for making data business-ready.”
Build the right talent and culture. “Data teams need to be prepared with the right tooling and processes,” said Prakash. “Analysts need to be enabled to use real-time data to generate insights. Decision makers need to drive right business decisions at the right time to make real-time analytics worthwhile.” In addition, he added, it’s important to be able to determine the extent to which data needs to be real time. Not every application requires subsecond data refreshes and analytics.
Strengthen your data governance. “The biggest issue isn’t technical. The biggest issue is trust,” said Kurt Muehmel, everyday AI strategic advisor at Dataiku. “Think about it this way: If you can’t write SQL yourself, you probably can’t read it well enough to know if the generated code is actually answering your real question. Sure, an agent might create a gorgeous dashboard in seconds, but is it showing what you actually need? Is it pulling from the right tables? Did it apply the correct business logic? Without proper verification, real-time generation just means we’re making mistakes faster. Companies need strong governance that clearly defines who gets to use these tools and on which datasets. Enterprises require transparent systems where you can trace every decision, not mysterious black boxes.”
Augment legacy systems. “Organizations should use techniques to augment core legacy systems—mainframes, traditional relational databases—by augmenting legacy systems with modern real-time systems that can deliver high performance at scale without compromising on consistency,” said Srinivasan. “This is referred to as ‘hollow the core’—preserving the existing systems for compliance and batch applications while moving all of the interactive customer-focused mobile and real-time applications to a modern, real-time platform.”
Centralize data functions. A unified approach to data management prepares organizations to be successful with real-time analytics, Streit advised. Look at approaches such as modern industrial data fabrics, as “they enable data intelligence across the business by centralizing data access and reducing the number of encrypted data pathways.”
Don’t try to boil the ocean. “A good way to do that is to clearly define your desired outcomes and identify the most critical data and business processes that will deliver immediate value,” Rolo said. “Build these foundational pieces first by creating efficient data pipelines for specific purposes and outcomes. Once these initial building blocks are working the way you want them to, expand incrementally to additional parts of the business, applying the same targeted approach. For organizations new to real-time data processing, partnering with experienced professionals can accelerate this iterative process, providing guidance on where to focus efforts and how to optimize each phase of implementation.”