These are hard data management and governance decisions for enterprises to make, but they are making them—and acknowledging that patience and adaptability are key for the coming years as data technologies continue to evolve and change the landscape. The data lake will continue to prove itself and make a fundamental shift in enterprise architecture in the coming years. When you take a step back and watch the business and IT drivers, momentum, and technology development, you can see how the data lake will become an epicenter in enterprise data architecture. If you take two steps back, you will see how 2015 developments could begin the evolution that transforms the data lake into a “data operating system” for the enterprise, evolving beyond business intelligence and analytics into operational applications and further realization of service-oriented architectures.
What’s Ahead in Big Data
In 2015, the mainstream adoption with enterprise data strategies and acceptance of the data lake will continue as data management and governance practices provide further clarity. The cautionary tale of 2014 to ensure business outcomes drive big data adoption, rather than the hype of previous years will likewise continue. Hadoop is clearly here to stay and “inevitable,” and will have its well-deserved seat at the enterprise data table, along with other data technologies. While Hadoop won’t be taking over the world any time soon, principle-based frameworks (such as our own modern data platform) recognize the evolution of both data technologies and computing price/performance on modern data architecture. Besides the usual maturing and improvements overall and for existing big data tools, we predict some major achievements in big data for 2015 that we’re keeping an eye on.
For more articles on big data technologies and trends, download the Free Big Data Sourcebook: Second Edition
The Apache Spark engine will continue to mature, improve, and gain acceptance in 2015. With this adoption and the incredible capabilities that it delivers, we could start to see applications and capabilities beyond our imagination. Keep an eye for these early case studies as inspiration for your own needs.
With deepening acceptance and recognition of YARN as the standard for operating Hadoop clusters, open-source projects and existing vendors will port their products to YARN certification and integration. This will not only close the gap between existing data technologies to work with Hadoop clusters but more exciting will be to see data technologies port over to YARN so that they can operate and improve their own capabilities within Hadoop. New engines and existing engines running on YARN in 2015 will further influence and drive the adoption of Hadoop in enterprise data architecture.
In 2014, we saw mainstream companies requiring data management features such as security and access control. These first steps will be critical to keep an eye on during 2015 for your own company’s data management requirements. Our concern here is that the sexy high-performance world of Spark and improved SQL capabilities will get the majority of attention, while the less sexy side of security and governance will not mature at the same rate. There is significant pressure to do so with the mountain of mainstream adopters waiting, so we’ll keep an eye on this one.
Finally, our most exciting item to watch in 2015 will be Hadoop’s subtle transformation as business drivers move it beyond a primary write-once/read-many reputation to that of full create/read/update/delete (CRUD) operational capability at big data scale. The benefits of the Hadoop architecture with YARN and HDFS go well beyond big data analytics and enterprise data architects can start thinking about what a YARN data operating system can do with operational systems. In a few years, this could also redefine the data lake —or we’ll simply create another label for the industry to debate. Once big data, high performance, and CRUD requirements are met within Hadoop, enterprise architects will start thinking about the economies of scale and efficiency gained from this next-generation architecture.
About the author
John O’Brien is principal and CEO of Radiant Advisors. With more than 25 years of experience delivering value through data warehousing and business intelligence programs, O’Brien’s unique perspective comes from the combination of his roles as a practitioner, consultant, and vendor CTO in the BI industry. As a globally recognized business intelligence thought leader, O’Brien has been publishing articles and presenting at conferences in North America and Europe for the past 10 years. His knowledge in designing, building, and growing enterprise BI systems and teams brings real-world insights to each role and phase within a BI program. Today, through Radiant Advisors, O’Brien provides research, strategic advisory services, and mentoring that guide companies in meeting the demands of next-generation information management, architecture, and emerging technologies.
In Q1 2014, Radiant Advisors released its Independent Benchmark: SQL on Hadoop Performance that captured the current state of options and widely varying performance. Radiant Advisors plans to release the next benchmark 1 year later in Q1 2015 to quantify those efforts.