Are today’s data systems — many of which were built and designed for legacy systems of the past decade — up to the task of moving information to end users at the moment they need it? And is this information timely enough? In many cases, there’s a lot of work that still needs to be done before real-time information, drawn from multiple sources, becomes a reality.
A new survey of 338 data managers and professionals who are subscribers to Database Trends and Applications reveals that real-time data access is still a distant pipe dream for at least half of the companies represented in the survey. The survey, conducted by Unisphere Research, a division of Information Today, Inc., in partnership with Attunity in March of 2012, finds that close to half of the survey respondents, 48%, report that relevant data within their organizations still take 24 hours or longer to reach decision makers. This suggests that much data is still batch-loaded overnight.
The trend is clearly moving away from batch, and a majority of respondents, 59%, are optimistic that timelines of the enterprise data they deliver will improve over the next 2 years. However, at this time, only 18% of their end users are completely confident in the timeliness of the data they are getting.
Such latency may have been acceptable in the days when business analysts worked tirelessly to build reports from data in different formats, then provided decision makers with a historical analysis of what recent promotions worked and where they worked, or parsed the preferences of customer segments from different regions. But more than a third of respondents say they would like their data delivery to be in real time.
Most companies are not yet data-driven organizations. The survey shows that organizations are still bound to traditional, closed, or siloed data management environments, thus impeding their ability to bring analytic capabilities to decision makers that need it. For today’s data manager or professional, the challenge is being able to provide end users access to actionable information in as close to real time as possible, at the push of a button. Today’s hypercompetitive economy demands that organizations be able to provide decision makers with the right information, at the right time.
However, moving this data to where it is needed may be a difficult challenge in many organizations. A sizable segment of companies, 61%, report they move only a fraction — less than 500GB — of data between systems and users on a daily basis.
There are challenges that still need to be overcome, the DBTA survey shows. Organizational and budgetary issues are the greatest challenges to achieving more timely data analysis. Almost a third of respondents agree that users want to see data closer to real time. However, funding to make this possible remains an obstacle, cited by 54% of respondents.
Then, there are the technical issues, starting with the fact that there is no one single source for relevant data; the typical enterprise today has a vast array of data types and formats pulsing through its veins, flowing in from its own systems and databases, as well as streaming in from external sources. Data quality is also a leading technical concern, cited by 40% of respondents. Companies now manage massive data stores, scaling into the hundreds of terabytes. (See Figure 4.)
Relational databases and data warehouses still lead as the primary platforms by which analytical data is delivered. The DBTA survey also finds that organizations are being deluged by big data, and an increasing proportion consists of unstructured data. Seventy percent of respondents to this survey also report that their levels of unstructured data—such as video, graphics, and web data—are rapidly growing.
The growing diversity of this data presents new opportunities and challenges to data integration initiatives. Most are attempting to address these new environments with traditional data management tools and platforms, but increasing numbers are now looking to virtualization and the cloud as well. Respondents recognize the advantages cloud computing can bring to real-time data integration, and many efforts are now underway, mainly with private or hybrid cloud-based services. More than two out of three respondents, 68%, see cloud-based data integration solutions as a way to improve and accelerate their organization’s data integration. The high-volume sites identified in the survey tend to be more likely to be shifting to these newer types of approaches to data management.
To download a copy of the survey report, “From Batch to Blazing: Moving Data in Today’s Real-time Business, 2012 Survey On Data Integration Strategies,” go to http://resources.attunity.com/GetReport.