In only a few years' time, the world of data management has been altered dramatically, and this is a change that is still running its course. No longer are databases run in back rooms by administrators worrying about rows and columns. Now, actionable information is sought by decision makers at all levels of the enterprise, and the custodians of this data need to work closely with the business.
That's because, in the wake of the recent financial crisis and economic downturn, there's a push from both high-level corporate management and regulators to achieve greater understanding and greater transparency across the enterprise, Jeanne Harris, executive research fellow and a senior executive at the Accenture Institute for High Performance, and co-author, along with Tom Davenport, of Competing on Analytics and Analytics at Work, tells DBTA. "In many ways, I think the ultimate result of the financial crisis is that executives realized they cannot delegate analytics to subordinates; they can't view it as technology or math that doesn't really affect them."
Harris urges executives to heed the words of one of the world's savviest business investors. "Warren Buffett used to say that you shouldn't invest in any company you can't understand," she points out. "I think if the financial crisis has proven anything, it's that you shouldn't run a company you don't understand. Executives recognize they need to get a lot smarter about what the key assumptions are underlying their business, and be a lot more thoughtful about how they make decisions. That's going to drive us all to a much more analytical future, a much more thoughtful future. We're going to see people making decisions based on data, as opposed to trying to use data to justify a decision they've already made."
Analytics is one of the most business-driven changes to the data management discipline. DBTA gathered input from a range of industry experts of the new direction of data and identified the following trends.
Shift to the Business Side
The emphasis of data management is now on what data can do for the business, not what the business can do for data management. "The applications, the capability or the competency of the people that are involved in data management have changed from a ‘techie' side to a business side," Rich Cohen, principal with Deloitte Consulting, tells DBTA. "We don't just have DBAs anymore who just worry about physically putting things together on a database. We're now more concerned with what it means and how you collect it. We're concerned about data governance with that. Ten years ago, I don't think anybody did anything more than just look at tabular or numerical data. Today, were looking at text, geocode images, sounds, and web. We can only use our imagination for what could be in the future as far as virtual data and virtual types of needs for data."
Expanding Use of Advanced and In-Database Analytics
Vickie Farrell, manager of marketing strategy for HP Software and Solutions, tells DBTA that advanced, actionable analytics has become "a high-priority trend for the BI (business intelligence) and data management industry." Data warehouses may not be enough to manage this new demand, she adds. "Right now, it is a common practice to extract data from an enterprise data warehouse and insert it into analytic data marts for advanced analysis. But adding an additional layer to the data architecture will increase complexity and potentially increase the time to make decisions. As a result, technology providers have started to push advanced analytics computation closer to data-and the goal is to continue that push."
Jinesh Radadia, partner for information management at EMC Consulting, says the push to incorporate greater analytic capabilities within enterprise data warehouse environments is giving rise to in-database analytics, which "supports more pervasive embedding of predictive models in business processes and mission-critical applications. With in-database analytics, enterprises migrate their predictive analysis, data mining, and other compute-intensive analytic functions from separate, standalone applications to execute in the enterprise data warehouse. Doing so allows leverage of EDW's full parallel-processing, scalability, and optimization features."
Many observers concur that analytics represents the greatest push in the industry. "The analytics world is more exciting now than we have ever seen," agrees Joe Lichtenberg, director of product marketing at Syncsort. "Companies are extracting more and different information from the data than ever before, new data visualization products are emerging. It's a great time to be a consumer."
Enterprisewide Data Integration
One of the most significant changes in recent years has been the embrace of data as an enterprisewide resource. "For many years, data warehouse and BI environments were built one application, one report or one data mart at a time," notes Farrell. "Project budgets didn't allow for a holistic approach to data integration and it usually wasn't necessary. But recently, leading organizations have begun to employ a coordinated, enterprise-wide approach to data integration, enabling cross-functional analysis and enterprise-wide performance management, and improving applications such as customer and risk management-replacing multiple isolated redundant projects."
Borrowing From ‘Green' Philosophy
Daniel Teachey, senior director of marketing for DataFlux, tells DBTA that Data Management 2.0 is based on the same principles as green IT-with the emphasis on "reduce, reuse and recycle." These approaches are essential in a tough, competitive economy, he explains. "Reduce" means to streamline "the number of enterprise applications that an organization has to manage," he explains. "This is critical as companies attempt to modernize operations on more cutting-edge platforms. For IT staff, a streamlined application footprint helps drive down the costs of supporting business systems. For the business side, fewer applications mean less data spread throughout the organization. This has the benefit of cutting down on confusion while also empowering line employees to make smarter day-to-day decisions."
The "reuse" aspect means moving proven data management strategies forward. "The rules, processes and procedures for collecting, managing and archiving data can be created once and used repeatedly," Teachey says. "This has the effect of making data management an instrumental part of daily business-and can increase the consistency, accuracy and reliability of data throughout the enterprise." Finally, "recycling" indicates "that the entire data management program can be thought of as recycled ideas from past efforts in other areas," he adds. "Ideas like Six Sigma and Total Quality Management have been popular ways for introducing quality into any business discussion. Why not recycle those ideas and extend them to the information that drives business?"
‘Big Data'
Syncsort's Lichtenberg calls this emerging era the era of "Big Data." Organizations are now charged with managing and extracting information from databases that are growing into the multi-petabyte range. "Coupled with ‘Big Data' is the need-and rapidly growing ability-to extract lower latency information from these massive and growing data sets," he tells DBTA. "Retailers are enriching the information in their customer data warehouse with years of click-level web activity for each customer. Banks are reducing data latencies for their production loan assessment and approval applications from intra-day to sub-one hour latencies." Many of these challenges are being met by high-performance data warehouses, Hadoop implementations, extreme performance, or data integration technologies. "It's a hot topic-and rightly so-as it's enabling the business to understand the data and offer new services in ways that were simply not possible a few years ago," Lichtenberg says.
Add to this mix a growing volume of "event" data, or data about operations, which is swamping storage systems and data warehouses. "These escalating data volumes are being driven by compliance requirements," Joe Gottlieb, vice president of marketing and business development for SenSage, Inc., tells DBTA. However, he adds, "this event data offers tremendous value and insight into the organization. Much like the data stored in data warehouses can be analyzed by business intelligence tools, event data can be analyzed to provide security intelligence and decision support for the enterprise, improving and strengthening security and IT operations."
Rise or Repurposing of Enterprise Data Warehouse
Data warehouses have dominated the data management and analytics space for many years, and many experts see their role as only increasing in importance. "They obviously aren't going anywhere," Jason Tiret, director of modeling and architecture solutions for Embarcadero Technologies, tells DBTA. "You may see smaller organizations leverage a federated approach with specialized data marts instead of a monolithic enterprise data warehouse, but for larger organizations, there is no way around not having an enterprise data warehouse. Organizations just can't fly blind these days with things changing so quickly."
Enterprise data warehouses will only become more critical as organizations increasingly rely on advanced analytics. "Advanced analytic models are key to the predictive enterprise," says EMC's Radadia. "Companies are finding out that it only makes sense to execute them natively inside the EDW, the chief repository of analytic information and horsepower."
However, not everyone shares the view that data warehouses will continue in their current form. Perry Rotella, senior vice president and CIO of Verisk Analytics, sees a shift toward more distributed forms of managing enterprise data. "Large data warehouses have proven to be complex to build and expensive to maintain," he tells DBTA. "We will see a movement from large-scale, enterprise data warehouses toward a technology framework that enables data sharing across business lines. The demand for a high degree of business agility and speed to market will drive data companies away from massive data warehouse projects towards enterprise data sharing frameworks."
More Data Appliances
One such evolution from the more centralized, Big Data model is the growing popularity of pre-configured data appliances that offer a quick on-ramp to analytic capabilities. These appliances typically prepackage and pre-optimize the processing, storage, and software components for fast OLAP and fast data loading. "These devices cost-effectively offload high-volume query processing from the EDW, provide a modular stepping stone to EDW-scale analytic workloads and play well with customer investments in BI and analytic applications," EMC's Radadia tells DBTA. "Devices offer a compelling value proposition to buyers of being easier to procure and potentially cheaper to deploy, as they require only one vendor to deal with and fewer resources for upfront integration and optimization due to pre-configuration by the vendor. They are moving into the mainstream of enterprise computing and networking."
More Alternative Databases
While the database market has always been in a state of constant change, the relational database management system vendors have tended to dominate the market. Lately though, there have been rumblings of movements to lighter, more web-friendly environments, such as the emergence of a "noSQL" class of databases. "Web applications and cloud computing are growing in dominance, both of which have architectures that are ‘scale-out' in nature," James Phillips, co-founder and chief strategy officer of NorthScale, tells DBTA. "The relational database management system, the default choice for holding the data behind web applications, is a ‘scale-up' technology-which means to handle more users, one must get a bigger server-adding CPUs, memory or IO capacity."
Along with noSQL, Phillips sees a range of lighter, faster database types that have increased in popularity, such as AltDB systems. These systems "have emerged in response to this scale-up versus scale-out mismatch. These technologies are designed to scale-out rather than up, are most appropriate for storing the types of information produced and consumed by web applications, and are optimized for the access patterns and consistency vs. availability trade-offs that characterize these software systems."
More Open Source
As described above, organizations are developing insatiable appetites for analytics. However, budgets aren't keeping pace with this demand. "Over the past 5 years, as the volume and complexity of data have gone up, budgets have gone down," Pentaho CEO Richard Daley tells DBTA. "At the same time, business owners, rather than IT, now typically control budgets for data management projects, making it increasingly difficult for IT to get approval for those projects unless they can clearly sell the value. Over the last couple of years, open source has become a mainstay for data management projects because if properly implemented, it delivers very fast time to value."
Rise of Services
A service-oriented approach-in which functions or processes are surfaced as standardized services capable of running anywhere within or between enterprises-is also reshaping to data management space. Initiatives such as service-oriented architecture and software as a service have caused "data management to change quite drastically over the last 5 years," says Embarcadero's Tiret. "These initiatives have forced organizations that didn't previously focus on governance to realize it is a necessity, not a nicety. They have to know where certain data lives, how it is used and how it is controlled and secured-and they need to know enterprise wide."
Cloud or Database as a Service
With the rising costs of maintaining databases and applications on premise, expect to see more database-as-a-service (DaaS) offerings and initiatives, Michael Kirven, co-founder and principal, Bluewolf, tells DBTA. DaaS can even be supported by cloud providers such as Amazon Web services, he continues. "In the next 2 to 5 years, we will see an ever-growing shift to technologies such as cloud computing and virtualization, helping to shape this new IT landscape-and DaaS enables CIOs to leverage these technologies."
Semantic Web
One initiative that is slowly becoming part of data management is the semantic web, in which actionable information is extracted from the web. With semantic web capabilities, organizations are better able to "reconcile and normalize meaning across data and content from diverse resources," HP's Farrell points out. "Also, semantic technologies will help computers glean implicit knowledge from explicit facts."
Adoption of semantic web approaches has been slow-going, however. "For a while, it looked as if widespread adoption of standards would enable the semantic web, but that is still a ways off as the industry still struggles with what it really means to be completely WS-* compliant," Lichtenberg says. "It will be interesting to see whether Hadoop, grid, and other high-performance tech-nologies provide us with an alternative approach to achieve this, as this approach doesn't require widespread standardization. There are probably some smart people working on that somewhere right now."
Bill Roth, chief marketing officer at LogLogic, agrees that "enterprises will turn to the semantic web slowly." He sees the first area of investment as "building out the ability to do ‘semantic equivalence'-such as determining that ‘Bill Roth' is the same as ‘William G. Roth,' or that ‘username' is the same as ‘usr_name.' This technology will develop in about three to five years. The core technology has been developed, like the ability to build ontologies, but they have not been widely adopted."
However, while enterprise adoption of semantic web approaches is tepid, Farrell sees adoption of semantic web approaches picking up in the IT vendor community. "Commercial application of semantic technologies is growing," she observes. "Semantic technologies are being used to automate product reclassification, allow for accurate and consistent diagnosis and treatment across hospital management communities, and execute dependency analysis for both managing and reconfiguring software assets."
The Data Road Ahead
Accenture's Harris observes that one thing is certain: The business is going to be more involved in data management than ever before. "There's going to be more data, and it's going to be more diverse," she says. "There's going to be a lot more visibility on data management, and data governance outside the IT world than there has been historically. There's a growing awareness among business executives that they play a more important role in data stewardship."