There is no limit to the potential, business- building applications for big data, springing from the capability to provide new, expansive insights never before available to business leaders. However, the new forms of data, along with the speed in which it needs to be processed, requires significant work on the back end, which many organizations may not yet be ready to tackle. IT leaders agree that to make the most of big data, they will need to redouble efforts to consolidate data environments, bring in new solutions, and revisit data retention policies.
These are the conclusions of a new survey of 322 data managers and professionals who are members of the Independent Oracle Users Group (IOUG). The survey was underwritten by Oracle Corp. and conducted by Unisphere Research, a division of Information Today, Inc. (“Achieving Enterprise Data Performance: 2013 Database Growth Survey”). Survey respondents come from a wide range of organizations, from manufacturers to educational institutions.
While many organizations have made it a priority to capture and use big data, they are also discovering the costs of managing and storing all this data. There is a drive across the industry to introduce new and more digitally compact forms of data storage, as well as cloud storage, but these solutions do not get to the heart of the problem for enterprises—data needs to be managed more effectively, and tied closer to the business, from the start. Storage is not an insignificant part of respondents’ IT budgets. One-third of respondents spend between 11% and 25% of their budgets on storage, the survey finds. In addition, one out of seven respondents report spending more than 25% of their IT budgets on storage.
Fueling today’s rapid data growth is rising business demand at respondents’ organizations. Almost nine-tenths of respondents say they are experiencing year-over-year growth in their data assets, and for many, this growth is in double digit ranges. Forty-one percent of respondents report data growth levels exceeding 25% a year. Seventeen percent report that the rate of growth has been more than 50%. Respondents within the utilities and telecommunications sector are seeing the fastest data growth, with 43% reporting annual expansions greater than 50%. The services and retail sector follows with 22%.
A multiplier contributing to this growth is data duplication across organizations for various purposes. In most cases, data is duplicated three or more times. Much of this data needs to be kept active as well. A majority of respondents, 61%, say they have increased the proportion of data kept online in the past 5 years (versus moving to archived tape) to address requirements for increased information accessibility. A key challenge mentioned by respondents is the fact that it’s getting more difficult to dispose of data. This, in turn, is exacerbating data growth across enterprises.
More of this data is kept online for easy access, despite the additional resources and costs incurred. Forty percent of respondents retain data well beyond the 7-year legal requirement in order to meet compliance mandates as well as maintain data in the event of litigation. Twelve percent of respondents, in fact, say they keep their data “forever.”
Data managers say they are most concerned about increases in data variety, as well as database performance, and the need to control data management costs. To deal with these issues, most respondents are focusing on ramping up database performance and consolidation efforts. There are also a number of “smart” data growth strategies that are starting to gain a foothold in some enterprises. For example, about one-fourth of respondents say their organizations’ backup data is stored in a public cloud (storage resources managed by a third-party offsite service). Another 15% of respondents say they have moved to a tiered-storage approach. A handful of respondents, 7%, have put database compression solutions in place. Another small portion, 5%, say they have a formal information lifecycle management process in place, which may deliver a cost-effective approach to managing and eventually retiring data, versus storing it forever on active disks.
However, many enterprises continue to rely on brute-force—and ultimately more costly or ungainly—responses to big data challenges. A majority of companies, 52%, still attempt to manage data growth through hardware acquisition and provisioning, versus more advanced and efficient approaches such as tiered storage or data lifecycle management. Close to half also upgrade their server hardware or memory to keep on top of growing data stores. In addition, a majority of enterprises rely on tape for backup and archiving. Most are now seeking more automated approaches to better manage growing data volumes.