Migrating to Unity Catalog (UC) is becoming a popular decision for many enterprises seeking to enhance data governance, improve security, and boost their overall data management capabilities. Guidance on how to effectively embark on that migration—without draining time and resources—is crucial to an organization’s successful pivot to UC.
Offering their expertise on UC migrations, speakers from Databricks and Blueprint joined DBTA’s webinar, Achieving Unified Governance for Data and AI: Unity Catalog Migration Simplified, to explore the ways to accelerate, simplify, and automate the transition process.
Data and AI governance—though critical—is complex, stated Ketan Ganatra, solution architect at Databricks. Being able to discover and trust data and its machine learning (ML) models is paramount for every business looking to compete in today’s market. Yet, the proliferation of data sources and consumers introduces a multitude of business challenges, including:
- Fragmented view of the data and AI estate
- Disjointed tools for access management
- Incomplete monitoring and observability
- Lack of cross-platform data sharing
The answer, Ganatra explained, is expansive—yet unified—data and AI governance. Quoting IDC, “organizations are finally realizing the value of data as an asset that needs to be protected, managed, and maintained to increase asset value.” Similarly, within the space of AI, Forrester’s 2023 AI Predictions report stated that “AI is now an enterprise essential, and as such, AI governance will join cybersecurity and compliance as a board-level topic.”
This need for data and AI governance is where Databricks Unity Catalog excels, according to Ganatra. Comprising discovery, access control, lineage, data sharing, auditing, and monitoring capabilities, UC unifies data and AI governance, as well as visibility, for enterprise customers. UC additionally simplifies permission models for data and AI, powers monitoring and observability with AI technology, and offers open data sharing.
Though the potential boon from implementing UC is clear, its activation is a journey. “That is where an expert like Blueprint comes into play. They have the experience and the tools and the techniques to help you…get started on using Unity Catalog—getting you… to start consuming your datasets the right way,” explained Ganatra.
Shannon Lowder, solution architect at Blueprint—a partner of Databricks—emphasized that UC ultimately helps to optimize enterprise lakehouses by delivering a single pane of glass for data and analytics, eliminating duplicate or redundant data and compute as well as managing access with less effort.
Lowder explained that Blueprint helps organizations get started with UC through a three-step process, listed as the following:
- Blueprint gains access to your data storage—such as Hive Metastores, Notebooks, SQL queries, dashboards, pipelines, and models.
- Blueprint runs an assessment on your environment to determine the UC configuration necessary for your unique data estate.
- Blueprint executes the configuration through repeatable transformations based on the desired state for UC.
Though seemingly simple, Loweder noted that there are hidden details within each of these steps; fortunately, Blueprint is prepared to “work through that complexity with you,” offering the necessary guidance to complete a successful UC migration.
Michael Hallak, director of product sales at Blueprint, summed up Blueprint’s mission succinctly, explaining that “we [Blueprint] have a number of tools and accelerators that we produce to…make sure that more folks can have unfettered access to these capabilities, independent of their ability to engage with us from a project or team-based approach.”
Supplementing Blueprint’s mission, Hallak led webinar viewers through a live demo of how Blueprint can help organizations migrate to UC.
For the full, in-depth webinar, you can view an archived version here.