INFRASTRUCTURE ISSUES
As real-time data rises, organizations’ infrastructures may not be quite ready for real-time deployments. Adoption of real-time data infrastructure “is unevenly distributed today,” said Pierson.
“We often see gaps between more tech-focused industries, which tend to adopt real-time patterns earlier, compared to industries such as ecommerce and finance, which might take more time. Most organizations also have a myriad of data management technologies and cloud providers, creating complexity and complications when accessing and delivering data in real time. Governance and privacy regulations are another complicating factor.”
Many organizations face challenges in leveraging real-time data “due to the lack of proper platforms, deployment methods, and governance practices,” Kachelmuss agreed. “Without a robust infrastructure and well-defined data governance processes, organizations will struggle to capture, process, and analyze data in real time, limiting their ability to gain timely insights and make informed decisions.”
Organizations also “tend to rely on their legacy processes and deployment methods, which are very different when dealing with real-time data,” he added.
While many organizations have made progress in adapting their infrastructures to support real-time deployments, “there is still more work needed,” said Mishra. “Organizations need to invest in infrastructure and adopt technologies such as containerized application management, distributed computing across edge and cloud, and stream-processing frameworks. This will ensure scalability and governance and orchestrate the data as it flows between the edge and the cloud.”
Still, even for enterprises that have been moving to real-time capabilities, the time has come to deliver tangible business benefits on these capabilities. “While most companies provide real-time experiences to their customers, the challenge is making those experiences relevant and intuitive with the right data, at the right time, in the right context,” said McLarty. “This can only be achieved through strategic integration and automation. For example, a logged-in customer shouldn’t have to provide the chatbot with any identifying information. This reflects the company’s IT landscape being too siloed. They may be able to provide some real-time interactions, but they are not going to meet real-time customer expectations.”
While an organization’s infrastructure may be ready for real-time deployments, “any existing data must be clean and organized before it can attain real-time data outputs,” Schommer added. “Currently, organizations are faced with siloed data, with data spread across the organization lacking any unification. They need to integrate tools that allow them to do more with less, improving data quality and allowing for real-time analytics.”
Also paramount is achieving visibility into real-time data flows. Managers “need the ability to respond rapidly to shifts in the market and to make quick, timely decisions,” said Chopra. The process for achieving visibility with their data “should be automated, transparent, and agile.” At issue are “enabling responsible consumption and making their data teams more productive. This consists of storing data in open formats, breaking down data silos, ensuring the right level of access and protection, and tracking the full lifecycle of assets, including business and technical lineage.”
Important to this process is to “establish a sound data architecture practice focused on ease of discovery and productive, responsible use,” said Chopra. “Real-time data streams increase the velocity and volume of data collected, and organizations need to ensure they have the proper data platform foundation to successfully harness the values afforded by these technologies.”
Chopra also recommended embracing commodity object storage, open data, and table formats to enable the widest amount of consumption patterns without lock-in.
Not everyone shares the view that a complete infrastructure overhaul is necessary in every instance, however. “I think the realization that lots of organizations won’t need any kind of real-time system is going to grow,” said Fritchey. “Sure, all this edge computing is very much at the forefront right now. However, your average insurance management application, for example, just won’t need it. First, you need to step back from the hype to identify your needs around this type of data. Then, where appropriate, implement what’s needed. And then, yes, there will be some need for additional training for some IT pros to handle the unique challenges real-time data collection and consumption contain.”
REAL-TIME DATA AND THE DATABASE
While technology stacks may or may not be ready to handle real-time data flows, today’s generation of databases appear to be real-time ready. “Today’s databases have evolved to support real-time scenarios,” Mishra pointed out. “NoSQL databases, in-memory databases, and even the updated SQL database offer improved performance, scalability, and support for high-velocity data streams. Technologies like a message bus or stream processing frameworks complement the database offering to enable additional use cases.”
The current generation of databases “have come a long way and are more than capable of supporting real-time data processing,” Kachelmuss agreed. “In-memory databases, NoSQL databases, GraphDB, and streaming data platforms—to name a few—all offer the scalability, speed, and real-time capabilities needed for handling high volumes of data in real time.”
At the same time, he pointed out, “the limitations in supporting real-time data primarily stem from internal processes and governance within organizations. Many organizations struggle to do this with batch data, and when working with real-time data, this only amplifies these struggles. While databases have evolved to handle real-time data processing, organizations must focus on addressing internal processes, governance frameworks, and cultural aspects.”
Inaccuracy with data being stored and accessed is a significant risk to real-time streaming. “The issue is using and presenting the wrong data for a status update and personalized offer to the customer,” said McLarty. Drawing conclusions based on inadequate data is another risk, he added. “Using the wrong or not full dataset can erode customer trust. Real-time data streams improve the availability and velocity of data in an organization. However, the flow needs to be complemented with data governance tools to ensure quality and completeness. Workflow tools to provide appropriate filtering and contextualization, as well as integration tools to ensure we reach the right endpoints, will minimize overall risk as real-time data analytics continues to evolve.”
Increasing the speed of data decisions and the amount of data stored can stress data platforms “that do not have well-established foundations of data governance, privacy and security, data integration patterns, and data observability,” Chopra warned. “These key tenants allow organizations to enable responsible, trusted, and secure data access and enable data teams to build data pipelines more quickly. Concepts such as data observability focus on establishing an understanding for the health and state of data, allowing organizations to identify, troubleshoot, and remediate data issues in near real time. Focusing on operational data from data pipelines and complimenting data lineage knowledge can afford organizations better understanding and visibility into issues before their consumers discover and report them.”