One of the challenges of working with Hadoop environments has been maintaining the infrastructure for big data projects. That’s where cloud makes things easier and, increasingly, has served as the underlying infrastructure platform of choice for Hadoop initiatives. At the same time, not everything has moved to the cloud just yet for big data environments. Many IT managers expect to live in a hybrid environment. They are planning for multi-cloud data management to deliver business value and are also still relying on old-school approaches and manual tools to support their data environments.
These are some takeaways from a recent survey of 216 data managers fielded by Unisphere Research, a division of Information Today, Inc., in partnership with Radiant Advisors, and sponsored by WANdisco. The survey collected responses from data users already working with Hadoopor considering a Hadoop data migration to the cloud, including those who have previously migrated their Hadoop deployment (“2021 Hadoop-to-Cloud Migration Benchmark Report”).
The survey found that most IT leaders are pursuing cloud migration to lay the foundation for future business value creation, including cloud-scale analytics and data modernization strategies. The survey also advises that we should expect a significant wave of Hadoop data migrations. At present, 21% of enterprises in the survey now have Hadoop running in the cloud, and another 32% are either planning to move or in the process of moving Hadoop to the cloud.
At the same time, Hadoop-to-cloud migration projects can take a toll on enterprises. About one-quarter of respondents indicated their projects were on time and on budget. Close to one-third, 31%, said their projects went over budget. Additionally, 44% said their efforts took longer than originally planned.
The two biggest factors impacting the cost of overall Hadoop migration projects came from the complexities of handling on-prem data changes during migration and from acquiring the IT resources to perform and manage data migration. These two challenges were followed closely by the cost of custom code development and maintenance of data migration scripts/programs.
The survey revealed that the challenges due to handling custom code development and the complexity of on-premdata changes during the migration “most likely led to the 55% of completed projects being over-time or over-budget,” noted the survey report’s authors, John O’Brien and Lindy Ryan, both with Radiant Advisors. “This aligns to the leading concerns companies have when planning to migrate their Hadoop data to the cloud—business disruption due to downtime, hidden costs and complexities, loss of critical business data during migration, and custom code development. Utilizing a Hadoop data migration-to-the-cloud tool would have likely controlled costs while eliminating complexities and potential loss of data changes during migration.”
After their companies complete the Hadoop data migration to the cloud, 42% of respondents said they planned to maintain a hybrid environment for the next 1–3 years, while 36% of respondents plan to maintain it indefinitely. The remaining 21% of respondents plan to shut down their on-prem Hadoop as soon as possible following cloud migration.
There is considerable manual work associated with Hadoop onsite-to-cloud migrations. For recent and planned Hadoop migrations to the cloud, 50% of respondents planned to handle, or have handled, data changes during migration with the use of software to automate migration of data changes. In contrast, 33% of respondents planned to manually reconcile any data changes, and 14% of respondents opted to not allow data changes during migration. Only 3% of respondents have no plan to reconcile the changes.
When asked which cloud providers respondents were leveraging at the time of the survey, the three major public cloud vendors led the way with Amazon Web Services (48%) followed by Microsoft Azure (38%) and Google Cloud Platform (21%). Other cloud vendors selected by respondents included Oracle (12%), IBM (10%), and Alibaba (4%).