Are data infrastructures ready to deliver insights at the speed of thought—or, for that matter, at the speed of an AI prompt? For decades, batch processing was the mode of data delivery—and organizations built their analytical environments in which information was several hours, or maybe a day, old. Then, as the digital economy evolved, certain parts of the data infrastructure moved to near real time, meaning results to queries were delivered within a half-hour to several hours.
Now, the requirement is to be able to process and deliver data insights at the moment of impact, and data managers and their organizations need to be prepared to deliver. However, at this point, many data managers are not fully prepared to provide real-time data access in terms of streaming data available for AI.
AI, of course, is the catalyst pushing all these real-time requirements and initiatives to the fore. Whereas the primary goal of data initiatives within the past 2 decades was to support analysis, AI is shifting that priority to action. And this requires real-time data to support machine learning, generative AI, and AI agents.
Best practices for preparing for and leveraging real-time capabilities mean new ways of looking at an organization’s data real estate. The following are steps to take to achieve a well-functioning, AI-ready data enterprise:
- Look at what the business needs.
Not every application or data feed requires real-time access. But in the age of AI, many parts of the business need to be able to react and leverage insights from customers or internal operations in order to deliver capabilities. That’s why it’s important to work closely with business decision makers to identify where investments and time resources need to be applied. Tellingly, organizations are being overwhelmed with data, and it’s often difficult to decide what data is right for real-time business needs versus data that can be held back. “[M]ore data causes anxiety and lack of action, instead of better decisions,” Bernard Marr, author of Generative AI in Practice: 100+ Amazing Ways Generative Artificial Intelligence is Changing Business and Society, points out in his book.
- Look to the long-term effects.
Simply bringing in expensive solutions and dropping them into the organization, hoping to see overnight success—in this case, a real-time data enterprise—is the surest path to disappointment. “As anyone who’s been through an enterprise resource planning, customer relationship management, or core banking project will attest, it’s an expensive, multi-year undertaking to rip out the core of a firm and replace it,” according to Stephanie Woerner, co-author of Future Ready: The Four Pathways to Capturing Digital Value. Success in moving to a real-time enterprise requires “building a platform that provides reusable and modular digitized business services that can be accessed across the firm and externally by partners.” This platform mindset encompasses silo integration, automation, clean data, and efficiency.
The watchword in such efforts is “simplification” to reduce spaghetti architectures and complex legacy processes.
- Look at agile and collaborative processes.
There are several practices and processes that need to be part of a real-time architecture. DataOps, AIOps, and MLOps are key methodologies for managing the unfettered movement of real-time data. DataOps (data operations) brings a collaborative and automated approach to the movement of data from sources through pipelines to targeted applications.
It puts all the people working within the data realm—DBAs, data engineers, data scientists, data analysts, and end users—on the same page. MLOps (machine learning operations) is also a collaborative and automated set of processes intended to employ machine learning to data flows.
AIOps (AI for IT operations) employs machine learning to automate data and application management, as well as to maintain performance.
- Look at the architecture.
Architecture not only maps out the present, it also provides a path into the future—for at least the next 2–3 years. A real-time-ready architecture needs to be designed and implemented around several approaches, starting with APIs that can integrate application and event activity with AI agents, and edge systems.
Event-driven architecture, which has been an architectural feature for more than 2 decades, is finally seeing the light of day with the advent of AI, providing a capability of detecting, alerting, and autonomously remedying material events that may be happening too fast for decision makers to act. Microservices also need to be part of a real-time architecture, as they capture and break monolithic applications or systems into manageable, loosely coupled services that provide the flexibility to assemble and dissemble components as the business demands. Open source resources such as Apache Kafka and Apache Spark-based solutions for real-time streaming or Apache Flink also support traditional batch environments along with real-time streaming.
Rather than attempting to tear apart existing infrastructures at considerable cost and disruption to the business, consider cloud services and cloud-native environments, which offer a range of capabilities, from elasticity on demand to autoscaling. Major cloud providers include tools and platforms that can ingest and analyze large volumes of events per second from applications or devices, as well as the ability to stream potentially up to millions of events per second into data warehouses and data lakes.
When data processing occurs close to the source, network latency becomes less of an issue. The devices deployed at the edge or in the Internet of Things process data near its original source, be it kiosks in hotel lobbies or sensors controlling the flow of water pumps. This avoids the latency of data moving between sources and centralized environments or clouds. Use cases include predictive maintenance, which analyzes patterns or anomalies in machinery and either provides alerts to human operators or conducts a workaround to fix an impending problem. Autonomous vehicles and robots (sometimes one in the same) need to conduct millisecond analysis of events or surroundings to perform the tasks at hand.
The real-time data enterprise is here. The journey to real time, however, requires careful and deliberate planning and preparation. Many organizations have only just begun to prepare for this new era of sub-second intelligence.