One in four enterprises now regards real-time data as critical to their ongoing operations—and another one in four is actively preparing to introduce real-time data capabilities into their infrastructures. To get there, they are increasing their adoption of emerging technologies.
These are some of the results from a recent survey of 241 readers of Database Trends and Applications. The study, which was sponsored by Pythian and conducted by Unisphere Research, a division of Information Today, Inc., included responses from IT and data decision makers representing a broad sample of company types and sizes. Next-generation data technology initiatives explored in the survey include machine learning, data lakes, Hadoop, the cloud, and data warehouses. These distinct technologies are interacting with each other, converging and paving the way to data-driven enterprises (“Profiling the Data-Driven Business,” 2019).
IT executives and their business counterparts understand the importance of a strong data strategy and its value to their businesses, and most are starting to get the key pieces in place to drive transformation through next-generation technologies and processes. The increase in data lake adoption and machine learning, the high level of interest in real-time analytics, and the anticipated movement of both data lakes and data warehouses to the cloud all support the case that many enterprises are taking firm steps toward modernizing and delivering greater data-driven capabilities.
Notably, there is a strong shift underway to real-time delivery of data and insights to further boost intelligent enterprise strategies. Close to half of the enterprises in the survey indicated they are aggressively preparing for real-time data capabilities to further enhance their data platforms. Forty-nine percent see real-time (subsecond) analytics, not just real-time ingestion, as a vital piece of their data platform planning. The biggest use cases for real-time data requirements include the more timely delivery of reports or dashboards, as well as ensuring real-time data feeds to decision engines.
Adoption of machine learning has almost doubled over the past year, the survey also shows. Use cases span both internal efficiencies as well as business growth initiatives, but challenges with access to the right data and a lack of operational automation are being reported.
Among data managers in this survey, there is quite a bit of enthusiasm for this higher level of automation. Currently, 48% of respondents are using the technology—up from 25% in a similar survey conducted a year ago. Machine learning has become valuable as companies are dealing with vast and rapidly growing volumes of data and the associated challenges of finding value and drawing insights from that data. The appeal with machine learning is that the algorithms do the heavy lifting of figuring out what data matters.
As a result, machine learning projects are being propelled by a number of factors, from identifying cost-saving opportunities to detecting potentially fraudulent transactions. More than four in five respondents said that their machine learning projects are being advanced by the need for operational improvements, and a majority said it is playing a role in managing security and risk, customer retention, and revenue growth.
Machine learning has numerous benefits, but the biggest challenge is scaling it to meet enterprise business requirements. Only one in five enterprises said that they have an MLOps process in place, defined as a practice for collaboration and communication between data scientists and operations professionals to help manage and automate production in machine learning.
Close to three in four respondents, 74%, pointed to issues with operationalizing machine learning models and pipelines. More than half of respondents also cited fundamental data issues underlying machine learning models, pointing to challenges with data quality and getting access to the right data. With only one-third of respondents noting a lack of executive support, we can expect to see machine learning become a key driver of innovation as these data challenges are overcome.
Data strategies have become part of enterprise business planning, the survey confirms. Data-driven innovation is being embraced by almost every department in the enterprise, led by outward-facing departments, with line-of-business owners and marketing departments demanding the most innovation from data.
Analytics planning is being shaped by competing requirements, including reducing costs while delivering better reporting and analytics, and enabling machine learning. While data warehouses and data lakes continue to be popular, data analytics is moving to the cloud, even as concerns about data security linger. A majority of data managers are having difficulties finding the talent they need to build data-driven enterprises.