Newsletters




AI Spurs a Revival in Data Mesh and Data Fabric

<< back Page 2 of 2

OPERATIONAL TECH AND DATA SPRAWL

The increasing convergence shaping up between information technology and industrial, or operational, technology is another powerful use case for data fabric.

This involves “connecting data across systems such as SCADA, MES, and ERP,” said Schindewolf. “Instead of replacing existing systems, many companies are focusing on making existing data more usable across engineering, operations, and corporate teams.”

Multi-cloud and SaaS sprawl also is driving the need for data mesh and fabric. “Almost nobody runs one warehouse anymore,” said Kumar. A typical data environment consists of “a mix of lakehouses, operational databases, SaaS tools, and streaming systems. Fabric is attractive because it promises unified access without forcing a physical migration of everything into one place.”

Plus, there’s simply too much data, with too many use cases. “Central data teams have hit a ceiling,” said Kumar. “Domain experts understand their own data better than a central team ever will, and business units want speed that a centralized bottleneck can’t deliver. That tension is exactly why federated governance has become the dominant theme.”

As a result of these pressing needs, metadata management, lineage, and semantic layers “have gone from ‘nice to have’ to foundational infrastructure overnight,” said Kumar. It’s important to treat data “as a product, with an owner, a contract, and a documented SLA,” he added. “And governance has matured alongside it, where fewer people are pushing decentralization for its own sake, and more are landing on a federated model: one platform, one set of standards, many domain owners.”

ALIGNING WITH THE BUSINESS

Business-driven ownership is essential to today’s and tomorrow’s data environments.

It’s important to look beyond data mesh and fabric as technology implementations and look to their potential impacts on the business. “Most organizations aren’t trying to become a perfect example of data mesh or data fabric implementation,” said Thomas. “They’re trying to build an environment where data can be owned by the teams closest to it, governed consistently across the organization, and accessed without unnecessary friction.”

Data mesh shifts away from technology-driven ownership “to allow for each individual business domain to take responsibility for its own data,” said Jason Ganz, senior manager for developer experience and AI at dbt Labs. This “offers clear demarcation that centralized architectures often lack. Data quality concerns in the age of AI are accelerating adoption. Enterprises need to bring their analytics systems closer to their operational systems and organize by domain.”

Iurii Iurchenko, senior data engineer at Life Line Screening, has seen this need firsthand. “Earlier, I focused on building reliable data pipelines, thinking less about business challenges, assuming the initial business request was complete, with a goal as simply implementing it,” he related. “Now, I verify business requests and ask more questions, leveraging LLM prompting, before tech implementation. Businesspeople do the same, but vice versa. They try to understand the technical part using LLMs. As an engineer, I represent the fabric part when building the unified data platform. Business represents the mesh pattern, trying to get their data.”

Iurchenko related how while working at Playtika, an umbrella for more than 30 separately acquired gaming startups, he “noticed that each game department tended to have its own data platform, creating a kind of mesh pattern.” However, “they faced challenges with unification and reinventing the wheel across different departments. The solution was to build a unified data platform on top of these mesh platforms.”

The more independent the departments are, “the more of a factor it is for mesh implementation,” Iurchenko concluded. “On the other hand, when the primary challenge is gaps in the holistic data picture, a lot of the same work is done in different departments, strict governance rules—that’s the precursor for fabric pattern implementation. Fabric implementation unifies the data, but the trade-off is that the data becomes generic. The mesh pattern, in turn, gives business departments the option to shape their own vision of data.”

LOOKING AHEAD

Real time is becoming more of a priority within data environments, and data mesh and fabric have a key role to play here going forward. “Most mesh and fabric implementations remain batch-oriented, which is a liability, as operational use cases like observability and security analytics demand freshness measured in seconds,” said Cucchi. “Governance is also moving earlier in the pipeline. Normalization, schema flexibility, and security controls all need to happen at ingest, and now, low-cost object storage, along with next-generation compression, are targeting making long-term retention feasible.”

The agent-native data product is on the horizon. “That’s the design challenge most teams aren’t ready for,” said Safronov. “Everything built so far (schemas, access patterns, pipelines) was designed for humans querying dashboards. Agents have entirely different requirements, such as sub-second freshness and granular permissions. The organizations that pull ahead won’t necessarily be the ones with the most sophisticated architectures. They’ll be the ones who stopped thinking of domain data products as tables for analysts and started treating them as APIs for agents. Most teams haven’t made that shift yet. The window to get ahead of it is closing faster than it looks.”

AI ended up “being the Trojan horse that brings everyone back to doing good data warehousing work again,” Bolles said. “A lot of that foundational effort lost its shine once AI became the only thing anyone wanted to talk about, but organizations without strong data governance and stewardship are only going to get so far layering AI on top of what they already have. Over the next year, I expect a reckoning where companies realize they have to pay attention to how their data is managed before they can do the impressive things AI promises.”

<< back Page 2 of 2

Sponsors