In today’s digital world, the importance of data continues to rise as companies strive to remain competitive and successful. Organizations collect massive amounts of data on customers, buyer behavior, supply, inventory management, sales, customer service, and more. Success is now defined by a company’s ability to leverage data for technical and business decisions, which requires accessibility and usability.
As data management leaders forge the best path for their data strategy, the rivalry between data mesh and data fabric is often top of mind. The answer is neither one nor the other, but rather the combination of these concepts. The key is understanding their strengths and how to efficiently integrate them into your own company’s strategy.
What’s Involved
Data fabric centers around the technologies that power metadata-driven use cases. Gartner defines data fabric as an integrated layer of data and connecting processes, which “utilizes continuous analytics over existing, discoverable, and inferenced metadata assets to support the design, deployment, and utilization of integrated and reusable data across all environments ...” However, leveraging metadata is a formidable challenge because in modern data environments, metadata is everywhere: in databases, ERP tools, on-prem, and cloud locations. How can data leaders navigate that growing landscape of metadata intelligently?
Automation is key. Enterprises can address the growing challenges of metadata by minimizing manual efforts, which are prone to error. Data leaders are increasingly partnering with technologies that automate metadata analysis, delivery, and repurposing. Data fabric allows users to more easily identify, understand, and apply metadata and parent data to current objectives.
Data mesh flips data fabric’s product-and-tech-centric approach on its head. Data mesh centers around people and expertise and emphasizes delivering high-quality data into the right hands. But for what kind of data environment is data mesh ideally suited?
Most data users at large organizations are, at this point, familiar with data lakes, which store vast troves of data. With time, rampant collection, and lack of organization, it’s common for data lakes to transform into data swamps. Historically, those charged with maintaining data lakes were ill-equipped to manage them intelligently. Data lake managers must not only comprehend the needs of data consumers, they must also grasp the domain expertise of data producers. Without a mesh framework, your average data lake manager fails on both counts.
Data mesh inverts the data fabric model to emphasize domain-driven design and product-centric thinking. Under a data mesh framework, those closest to the data are responsible for the health of the data, delivering data as a product, and communicating objective measures, such as data quality.
Data Mesh and Data Fabric: A Perfect Pair
By now, you may realize that these two concepts are not mutually exclusive. Data mesh and data fabric are complementary and can be combined into a “meshy fabric” that helps overcome bottlenecks while ensuring data remains at the forefront. It is up to the data leader to determine when, where, and how to implement each framework into their own organization.
For example, organizations that constantly strive to understand and use data across departments will benefit from a data fabric. And those who want to help data engineers put quality data in the hands of data consumers will also require a data mesh. These goals are not always independent, and the best results come from combining both.
Wherever you decide to begin, it is essential to start by preparing people within the organization for this fundamental shift, where data producers collaborate more directly with data consumers. This will involve data-producing teams, data consumers, data engineers, and decision makers.
The Importance of a Data Catalog
After focusing on the people, it is time to pivot toward technology. The success of a “meshy fabric” requires a solid foundation: a data catalog. A data catalog is the catalyst and the glue that unites data mesh and data fabric. Modern data catalogs are no longer the data dictionaries of the past. A modern data catalog captures a broad range of metadata, including business intelligence and metrics, as well as key terms, domains, and functional processes. All together, this enables the data catalog to serve a much wider selection of consumers. The data catalog will measure human behaviors around the data, highlighting the most valuable and actionable data. It can then be used to empower data analysts, data scientists, and business users alike to use and understand that data.
Data Governance as Fuel
The next critical step, which is often overlooked by data and analytics projects, is data governance. Governance is the fuel that powers your entire strategy. With your data catalog acting as the foundation, a data governance program can more easily scale across the business. It offers a place to house crucial definitions in its glossary and defines the data domains that enable data consumers to quickly find what they seek. This is another reason why it is important to start with the people. At this point, collaboration is key—especially when setting up data domains and definitions. An organizational chart will be the guide for defining terms, collaborating, and aligning on shared definitions that all can agree on.
Enterprises that garner the most success are the ones most passionate about making sure all data becomes well-managed and documented. All in all, it may feel daunting to align people, processes, and technology around the marriage of data mesh and fabric, but by combining the best of decentralized autonomy, your organization will be able to move at maximum speed—with the right oversight and risk mitigation. It also creates a win-win for data consumers, who have a reliable source of well-defined data products with owners who are directly accountable.