The Battle Over Persistence is somewhat of a ‘holy war’ between the data consistency inherently derived from a singular data store and the performance derived from data stores optimized for specific workloads.
Shifting gears into a new era has never been easy during the transition. Only in hindsight do we clearly see what was right in front of our faces—and probably the whole time. This is one nugget of wisdom I have been sharing with audiences through keynotes at data warehousing (DW), big data conferences, and major company onsite briefings. Having been part of the data management and business intelligence (BI) industry for 25 years, I have witnessed emerging technologies, business management paradigms, and Moore’s Law reshape our industry time and time over.
Big data and business analytics have all the promise to usher in the information age, but we are still in the infancy of our next era—and frankly that’s what makes it so exciting!
In 2013, the marketplace for big data, BI, NoSQL, and cloud computing saw emerging vendors, adapting incumbents, and maturing technologies as each compete for market position. Some of these battles were resolved in 2013, while others will be resolved in later years—or potentially not at all. Either way, understanding the challenges on the landscape will assist with technology decision making, strategies, and architecture road maps today and when planning for years ahead.
Two of the more dominant shifts occurring around us can be called the Battle Over Persistence and the Race for Access Hill.
Big Data's Battle Over Persistence
The Battle Over Persistence didn’t just start 5 years ago with the emergence of big data or the Apache Foundation Hadoop; it’s been an ongoing battle for decades in the structured data world. As the pendulum swings broadly between centralized data and distributed disparate data, the Battle Over Persistence is somewhat of a “holy war” between the data consistency inherently derived from a singular data store versus the performance derived from data stores optimized for specific workloads. The consistency camp argues that with enough resources, the single data store can overcome performance challenges, while the performance camp argues that they can manage the complexity of mixed heterogeneous data stores to ensure consistency.
For more articles related to big data, download DBTA's Big Data Sourcebook.
Image of big data related items courtesy of Shutterstock.