The use of real-time data has become common in industries such as finance, retail, and healthcare. This has optimized everything from fraud detection to personalized online shopping, even ensuring that hospitals are staffed with appropriately skilled workers and supplies at the right time. Not only is the use of the technology spreading across industries and companies of all sizes and types, it’s evolving to integrate streaming event-based data with historical data to provide contextualized (and more intelligent) insight for actions.
The reality is, however, that most companies do not do much, if anything, with streaming data—at least not yet. According to IDC Research, only 26% of streaming data is analyzed in real time before it’s moved to a repository such as a data lake. And, while historical data is a valuable contextual component for decision making, data loses value as time passes.
Contact information changes, a person’s credit score rises and falls, a pandemic changes everything, and so on.
Companies that do not act based on new data as it streams in will soon be at a big disadvantage. The ultimate aim is to take action on new data as it is born, which is when it is most valuable, and to have that insight enriched with context from historical data too. This is now possible with unified real-time data platforms.
“There’s a growing appetite for streaming data in a wide range of industries and use cases because companies are under ever-greater pressure to enable rapid time-to-insight from numerous data sources and types,” concurred Jason Stamper, research manager at IDC.
But how does a company begin to re-architect its data infrastructure and reorient business processes to lead in what’s being called “the real-time economy”?
Finding the right real-time data platform depends on your organization’s specific needs, technical resources, and internal skill sets. It also depends on where your company wants—and needs—to take real-time data into the future.
It’s relatively easy to harness real-time data streams using stream processing capabilities. It’s much more challenging to set up a mix of systems that will intelligently unify streaming (data in motion) and historical data (data at rest) without losing real-time momentum and the value associated with it.
There are four basic questions that organizations should be asking themselves as they plan for systems moving forward:
- What are the potential roadblocks to getting that real-time data?
This will vary on a company-by-company basis. For companies in industries such as manufacturing, sensors are a source of real-time data. For services dealing with fraud prevention, real-time data from sources such as customer interactions and market feeds is very important. In general, data often flows into silos, so one part of the business might collect the data, but it doesn’t get shared with other parts of the business.
Where is your data getting stuck or hitting bottlenecks? Why is that happening? Do you have a good real-time data ingestion pipeline setup? Can your application access real-time events and data feeds before they land in a data warehouse or data lake?
- Are you streaming data only or processing and incorporating historical data?
Identify data that you can process in real time that will make a difference to what you want to accomplish. Also, identify real-time data you are collecting but not processing in real time. Collecting real-time data into a data warehouse or data lake for the sake of collecting data won’t net a meaningful or timely result. What data is coming in real time that could be converted to insight and action to address your company’s pain points or enhance the customer experience?
Also look at your customers’ negative experiences and identify data that could mitigate them. Each company will have different scenarios, but here is one example of real-time data that has a real-world impact. Imagine someone shopping at a physical store. A good salesclerk might recommend a shirt that goes well with the sports coat being purchased and knows that the customer likes such shirts because they are wearing something similar. That’s new insight informed by new data (the sports coat purchase) and historical data (the current shirt). In a digital economy, that same scenario happens as seamlessly online as in the real world.
- Build or buy, prepackaged or open platform?
There are prepackaged applications that you can potentially customize. Then there are data platforms and services that are open and flexible to build your solution on top of. Some open data platforms allow for ingesting from relevant data sources and support the purposeful combining of data in motion and data at rest. Few companies today have the technology resources or in-house skill sets to build such data platforms from the ground up. Many data leaders recommend leveraging open platforms that allow them to change and modify their applications as business requirements evolve.
- What model is best: a self-hosted or managed service?
Self-hosted and managed service models offer, respectively, a decrease in levels of system responsibility. Additionally, a serverless computing model can further abstract away the operational complexity of the system. Amongst other aspects, the choice would depend on your technical talent.
How much of their time is most effectively spent on dealing with the infrastructure versus building the business application? Because wiring real-time services and capabilities is a complex venture, and because the technology is virtually a greenfield for many industries (and certainly many organizations), the managed service and serverless model may be the best way to start. Most managed services have consumption-based pricing, so companies pay only for what they need and the resources they consume, enabling them to start small and expand as they discover new ways of using the technology.
And, without worrying about server issues, upgrades, or any other operational tasks, developers can focus on building applications that take advantage of real-time capabilities and build value for the organization.