While the emerging constellation of next-generation data architectures—fabric, mesh, and cloud—is extremely appealing, it’s still full of unknowns. These approaches present opportunities for greater data democratization, but also increased complexity.
Understanding the distinctions between data fabric and mesh are also important before moving to this architecture.
Data mesh is a highly decentralized, self-service architecture in which datasets are managed or controlled by business units across enterprises. Data fabric is a more centralized architecture that supports metadata designed to integrate disparate, multiple data platforms and pipelines that simplify access to these assets. “Data fabric emphasizes virtualization and centralization of data to create a unified data infrastructure by integrating different data sources located in different systems and different cloud environments,” said Anil Dangol, data manager at Launch Consulting. “Data mesh, on the other hand, emphasizes decentralization of data which advocates data as a product where each team owns the data product.”
Both fabric and mesh “are reasonably early in their technology evolution,” Jim Webber, chief scientist at Neo4j, pointed out. A survey of more than 200 IT leaders by Unisphere Research, a division of Information Today, Inc., finds cautious uptake of these modern data architectures. While fewer than 1 in 5 enterprises are using some variation of data mesh and fabric architectures, many are cautiously eyeing the technologies (“The Move to Modern Data Architecture: 2022 Data Delivery and Consumption Patterns Survey,” May 2022).
Cloud-based architectures, conversely, are relatively mature, and are delivering greater capabilities. “Almost all organizations that are building a new data landscape are leveraging cloud-based technology,” said Steve Jones, vice president of insights and data at Capgemini. “Even traditional extract, transform, and load-based architectures benefit from the power of cloud to enable dynamic capacity to speed up processing without requiring large-scale continual infrastructure.”
Data “is the core of digital enablement and most organizations today are moving away from a one-size-fits-all data strategy to a more modern platform that focuses on enabling data products, focused data stores, machine learning, or AI solutions, and other data services,” said Pranabesh Sarkar, data architect at Altimetrik. “The focus is on building an integrated data ecosystem leveraging a foundation built on data lake, mesh, or data fabric.”
Moving to data fabric and mesh requires extensive rethinking of data management and analytics processes, Jones explained. Companies are “still learning how to transition from passive, post-transactional data architectures to ones that support a data-driven organization. This isn’t a question of architecture as much as it is of governance and culture. Some organizations have made the technical shift of moving toward a data-mesh infrastructure, but without the associated culture change, they’ve ended up in a similar place—just using different technologies.”
Building out a data fabric may be an easier transition than mesh. Currently, “enterprises are leaning more into fabric so that they can master their data,” said Webber. “This is a common pattern we see at Neo4j with metadata knowledge graphs. Mesh seems to be for those of a more adventurous mindset with teams offering their data into the mesh for others to consume.” The most compelling use case for mesh at this time, Webber added, is helping with “discoverability and reuse of siloed data.”
In recent years, there has been a lifting and shifting of data into the cloud, which has created its own problems.
“However, with this rapid migration there were a lot of data design patterns involved which required copying the data into multiple different places,” said Dangol. “It did not fully address data as a product creating a lot of challenges, such as data governance and security, storage, quality, and data life cycle management. Data mesh and data fabric try to address those problems in their own ways.”
The appeal of data mesh stems from “treating data as a product, which pushes data ownership responsibility to the team with the domain understanding to create, catalog, and store the data,” said Mathias Golombek, CTO of Exasol. “Doing this at the data creation phase brings more visibility to the data and makes it easier to consume—and stops any human knowledge siloes forming. This opens up data democratization. Employees can focus on experimentation, innovation, and producing more value from the data. That’s the theory, anyway.”
ADVANTAGES REALIZED
Data fabric and mesh, built on cloud, may open up data processes in ways not possible before. “At a technical level, next-generation data architectures support better data discovery and access, leading to data democratization, simplified and agile data flows, faster time to insight and time to value, and the ability to industrialize the application of AI,” said Naveen Kamat, executive director and CTO of data and AI services at Kyndryl. “At a business level, this can mean you are now enabling a whole new set of business outcomes from customer experience to productivity and revenue maximization.”
The rapid availability of data through next-generation data architectures means more rapid business responses. “Operational speed data will improve business outcomes where insight at the point of action drives better business performance,” said Jones. “This is the tip of the spear for next-generation data architecture, looking at where data-driven applications exist, as opposed to applications simply dumping data into data stores. Then look at who benefits, and therefore who is accountable for delivering those benefits. Above all, focus on the cultural change you want to achieve.”
As they advance at their individual paces, these next-generation architectures have become crucial components of budding data modernization efforts—of which only about 20% of companies have completed, said Bret Greenstein, partner for data, analytics, and AI with PwC. “Initially, 3–5 years ago, companies were adopting cloud by lifting and shifting the legacy data systems they had into cloud as-is. This was a fast approach, but it didn’t do anything to enable new business outcomes or to simplify and speed the flow of data through the enterprise. However, in the last several years, the more strategic approach of data modernization has become the dominant pattern for reaching next-generation data architectures. This approach leverages data mesh principles, on cloud, designed in ways that maximize business value and usually create dramatic simplification.”
Overall, decentralized data creation “brings more visibility and makes data easier to digest and consume,” said Golombek. “It also helps to truly democratize the data because data consumers don’t have to worry about the data discovery and can focus on experimentation, innovation, and generation of new value from data. Because of the decentralized data operations and the provisioned data infrastructure as a service, data mesh results in greater agility and scalability, with teams focusing on relevant data products. It also supports the creation of a federated, global governance that enables interoperability and simplifies access to data.”