Data Fabric for the Bigger Picture
The concept of data fabric architecture has been around for some time, and now, in the age of data-driven enterprises, “it holds significant potential in transforming the landscape of data management and analytics,” said Rogers Jeffrey Leo John, co-founder and CTO of DataChat. “The primary advantage of the data fabric architecture lies in creating a unified view of enterprise data for AI, which encompasses both business logic and the actual data across an organization.”
For example, an AI-powered Q&A system “must comprehend the business logic and organizational context,” Leo John explained. “Today, such AI systems must interact with and combine information from diverse information sources such as knowledge graphs or semantic layers for business logic, as well as data warehouses, data lakes, and document stores for actual data. Additionally, integration with data observability and metadata management platforms is necessary for compliance and governance.”
By integrating these components into a unified view, data fabric helps “enterprises accelerate innovation in their analytics and AI initiatives, enabling swift business decision making and the creation of AI-powered experiences for both internal and external users,” said Leo John.
At the same time, implementing a data fabric architecture can be challenging, “as it demands the appropriate expertise in both data and IT skills for setup and management,” he said. “It’s not merely a technological change; it also requires consumers of data within an enterprise to reconsider their approach to data. This shift in mindset may present a barrier to adoption, particularly if the existing data processes are intricate, and if there are existing tools within the enterprise that do not integrate seamlessly with the fabric architecture.”
Vector Databases
The rise of AI and GenAI calls for databases that can handle and prepare unstructured data for the LLMs that are driving new forms of intelligence. “AI tools like ChatGPT are built on top of vast amounts of unstructured data,” said Andre Zayarni, co-founder and CEO of Qdrant. “However, developing such tools with the help of large language models is extremely resource intensive. Training such AI tools with specific knowledge requires immense amounts of compute power and memory.”
A vector database “can drastically reduce an LLM’s resource requirements,” Zayarni continued. “Vector databases are making data management more efficient, by indexing, storing, and retrieving immense amounts of data at a fraction of the cost.”
The secret to dedicated vector databases, he explained, “is that they use ‘approximate- nearest-neighbors’ algorithms which are able to determine the most semantically relevant information much better than the LLMs themselves. By eliminating the need for extensive preprocessing, this new approach simplifies management and extracts more value from data that was once difficult to use, compare, or contextualize.”
Vector search, the core capability of vector databases, “unlocks a wide range of powerful applications,” Zayarni said. “Advanced search capabilities deliver more nuanced, context-aware results beyond basic keyword matching. Anomaly detection is improved by comparing data points in vector space, making it easier to spot outliers. Retrieval-augmented generation, or RAG, uses vector search to dynamically pull relevant data, enhancing natural language processing tasks. Additionally, recommendation systems powered by vector search provide highly personalized suggestions based on similarity in vector space, significantly improving user experience and engagement.”
The main challenge to vector database adoption at this time is “a lack of familiarity with the technology and a deep understanding of how to use vector search effectively,” Zayarni said. “Initially, introducing vectors and the sophisticated techniques for data storage and retrieval might seem daunting. However, as organizations begin to experiment and understand these technologies, they discover that these features are manageable and offer significant benefits.”
Systems of Intelligence for Actionable AI
There has long been a gap between established systems of record and systems of engagement in enterprises, creating data management and integration headaches.
Now, the time is ripe for systems of intelligence, or SOIs—which are emerging as the intelligent link between these two data environments.
“The SOI takes data from the transactional systems, generates actionable insights and recommendations, instructs the systems or record and engagement to take an action, and then finally learns from the outcomes of those actions with speed and at scale,” said Stephen DeAngelis, founder and CEO of Enterra Solutions. “In the past, SOIs were out of reach due to technological constraints, but the rise of next-generation technologies such as generative AI and autonomous decision sciences has rendered them achievable.”
SOIs integrate with an organization’s current system of record and system of engagement, “blending human-like reasoning and generative AI with transparent glass-box machine learning and nonlinear optimization,” DeAngelis explained. “Acting as the company’s centralized brain, it automates the analysis of intricate datasets to produce prescriptive and anticipatory insights regarding supply chain planning, consumer preferences, and revenue growth opportunities driven by changing demand patterns.”
For example, “supply chain professionals can become equipped with the critical insights they need to perform their job better or overcome disruptions,” DeAngelis said. “SOIs can also offer automated recommendations on optimized pricing, trade promotions, media mix, price pack architecture, and product assortment for a wide array of products, helping sales departments strategically price, promote, package, and assort their products to best meet consumer demand.”
The challenge is that “SOIs rely on data sourced from diverse channels like enterprise resource planning and customer databases—system-of-record data—to social media platforms and customer relationship management systems—system-of-engagement data,” DeAngelis cautioned. “Establishing robust data foundations and achieving data maturity [are] critical. Companies must ensure their data infrastructure is robust, secure, and cloud-based for seamless accessibility and integration.”