Implementing data fabric architectures has circled many conversations that often arise when examining how to unify data across disparate sources, creating a coherent data service. With the explosion of generative AI (GenAI), the notion of a data fabric has grown greater in importance, offering a potential avenue toward using natural language to access data and analytics, improve the data itself, and reduce labor-intensive data cleansing.
At the annual Data Summit conference, Jeff Fried, director, platform strategy and innovation, InterSystems, led the session, “Taking Data Fabric to the Next Level,” discussing how a mix of a data fabric architecture and GenAI can enhance data management practices, offering technological examples, real-world scenarios, as well as risks in execution.
The annual Data Summit conference returned to Boston, May 8-9, 2024, with pre-conference workshops on May 7.
Fried defined data fabric as an architectural pattern and integrated layer of connected data that enforces centralized governance.
Adopting data fabrics has risen in popularity because most organizations “have a hunger for data that is insatiable, yet a capacity to control data that is limited,” explained Fried.
Other drivers for implementing a data fabric architecture include enabling less technical users to quickly find, access, integrate, and share data, as well as generally reducing the cycle time of accessing ready-to-use data—to name a few.
Smart data fabrics, as coined by InterSystems, apply a variety of functions between the data source and the consumer, including integration, security, data exploration, BI/analytics, normalization and harmonization, and more. A real-time smart data fabric connects dozens of applications—regardless of its dissimilarity—that synchronizes data, calculates real-time positions with “on-the-fly” aggregation, and remarkably enhances performance without using more of the infrastructure.
Fried warned that adopting a data fabric is not simply adopting any sort of product or technology—yet there are ways to ease implementation.
“You won’t be able to find a ‘data fabric in a box’ anywhere,” noted Fried. “Yet, you can create pre-generated patterns that allow you to quickly generate value.”
Generative AI (GenAI) certainly has a role to play in the data fabric role, according to Fried. Appropriately, when asking ChatGPT, data fabrics offer GenAI:
- A unified architecture for data integration that the GenAI uses to train models effectively
- Centralization of accessibility and governance that ensures that GenAI has access to secure, compliant data
- Horizontal scalability that allows GenAI to train models faster and more effectively
- Enables GenAI to ingest and analyze data in real time
However, your data is probably not AI-ready. A smart data fabric can help achieve this readiness, enabling organizations to:
- Handle data at scale, incorporating more data from more sources
- Transform and unify data through harmonization, cleansing, and consistency to make data healthy for AI use
- Help control what is fed into large language models (LLMs) and how
There are three prevalent patterns of a combined GenAI and data fabric implementation, according to Fried, which include creating natural language assistants with the data located in the data fabric, embedding GenAI into existing applications, and creating custom coding assistants.
Ultimately, Fried urged listeners to look for data platforms that incorporate analytics, vectors, and LLM integration directly within the data fabric. This will enable:
- Elimination of latency, data duplication, and data consistency issues
- Easier implementation of GenAI-enabled apps
- Faster time-to-value, simpler operations, less complexity, and lower TCO
Many Data Summit 2024 presentations are available for review at https://www.dbta.com/DataSummit/2024/Presentations.aspx.