Big data is here, offering both vast opportunities — as well as vexing challenges — for every organization it touches. For a number of years, it has been understood that to be of value, information needs to be readily available, as close to real time as possible, to users in any location. Now, with the onset of “big data,” the task gets more daunting. “These are all increasing the demands on both transactional and analytics data systems,” says Bernie Spang, director of database software and systems for IBM.
Yet, the enormous competitive advantages that come with effectively managing big data can far outweigh performance issues. Benefits include “providing more customized offerings tailored to the individual purchaser, better insight into consumer purchasing behavior, and customer acquisition strategies able to predict who is more likely to purchase,” says David Noy, director of product management at EMC Isilon. The problem is many existing databases and data environments are not ready for the onslaught. Performance may degrade as massive amounts of data in various formats are brought into current environments.
Access the full version of this article in the DBTA Thought Leadership section on Database Performance Technologies in the Era of Big Data. A short registration form is necessary to access the special section.
Many of today’s systems “don’t scale queries well, don’t have tools that scale, and have not yet embraced in-database technology enough to cope with the variety and volume of data,” says Dan Graham, general manager for enterprise systems at Teradata. “Even some of the [massively parallel processing] shared-nothing databases don’t scale in all dimensions. They may be able to store 500TB, but not able to handle 20 to 50 table joins, manage mixed workloads, and run in-database analytics well.”
Accordingly, a study of 298 companies (“The Petabyte Challenge: 2011 IOUG Database Growth Survey”), conducted by Unisphere Research, a division of Information Today, Inc., found that database performance was cited as the top challenge to maintaining the timely delivery of data. Almost all survey respondents report significant data growth from year to year. The most frequent response to this data growth — cited by two-thirds — has been to attempt to scale with new hardware purchases. In many cases, today’s IT infrastructures simply are not ready for the large amounts of data that need to be processed, managed, stored, or archived. Half of the respondents, for example, say data growth is currently outpacing storage capacity.
Along with the technical issues associated with managing data growth, many organizations are simply not ready from a management point of view. “Not only are today’s databases not ready for the big data deluge, but also 90% of today’s organizations are ill-prepared to handle today’s needs,” relates Peter Aiken, Ph.D., associate professor of information systems at the Virginia Commonwealth University School of Business. “Fewer than one in 10 organizations has a strategy for leveraging their data.”
What does it take, then, to ramp up performance to meet big data challenges?
This article continues in the DBTA Thought Leadership section on Database Performance Technologies in the Era of Big Data. A short registration form is necessary to access the special section.