The emergence of vector databases and vector search for handling massive quantities of complex data have radically transformed the way AI is implemented and managed. As a specialized approach for managing multi-dimensional data, vector technology proves to be an essential component of advanced AI applications.
Experts from Datavail joined DBTA’s webinar, Vector Databases: Innovating Data Management in the AI Era, to examine the nuances of vector databases and vector search, as well as its role for AI implementation.
Michael Agarwal, director and global practice leader, cloud databases at Datavail, started with the basics, defining a vector as an embedding that refers to a mathematical representation of data in a high-dimensional space. More simply, a vector is a list of numbers or coordinates that represents data when it’s encoded.
Embeddings measure the relatedness of text strings, where small distances indicate high relatedness and large distances indicate low relatedness. This technology is often used for:
- Search, where results are ranked by relevance to a query string
- Clustering, where text strings are grouped by similarity
- Recommendations, where items with related text strings are recommended
- Anomaly Detection, where outliers with little relatedness are identified
- Diversity Measurement, where similarity distributions are analyzed
- Classification, where text strings are classified by their most similar label
A vector database, Agarwal explained, stores, indexes, and manages vector embeddings—including images, videos, audio, and text—in a form that computers can process for search and retrieval tasks. If you have a vector embedding, you need a vector database.
A good vector database will be conducive to vector search, allowing you to search vectors based on semantic similarity. This search needs to be fast, as an enterprise may store millions or billions of vectors.
Jorge Anicama, practice leader, analytics at Datavail, pointed out that vector databases are what enable AI and large language models (LLMs) to infuse domain knowledge with question/answer-based AI. This functionality—known as retrieval augmented generation (RAG)—requires vectors to represent text embeddings that indicate a specific piece of data. These text embeddings, which become vectors, represent your domain knowledge.
Once these vectors are stored in the vector database, this information is made available through an application. When a user interacts with that app by asking questions about the domain knowledge, the question passes through the LLM and is converted into a vector. Then, this vector is going to search for a similar vector within the database through a proximity algorithm. The most similar vector found will, once again, be fed through the LLM to be converted back into text, creating a natural language conversation between data and user.
Srinivasa Krishna, director and global practice leader, MySQL and MariaDB, at Datavail, explained that as the demand for generative AI (GenAI) applications continue to surge, vector databases will play a key role in the space.
“Their [vector databases] ability to efficiently store, retrieve, and contextualize data in real-time makes them more appealing for these [GenAI] applications,” said Krishna.
He then divided vector databases into three types, including:
- Vector Libraries: Lightweight, open source, easily available, and can be integrated easily into app codes
- Specialized Vector-Only Databases: Store embeddings in a database and then make them queryable, mainly for specific use cases
- Enterprise Databases with Vector Support: Many large databases are expanding their functionality with extensions/plugins for vectors
Krishna then walked webinar viewers through different databases and their vector capabilities, such as:
- MySQL and MariaDB: Uses traditional database methods while offering special data types for vector store with vector search functionality. Also has several capabilities in private preview or that are under development, including Vector Store in MySQL Heatwave, MariaDB Vector Engine, Vector Search in Google CloudSQL for MySQL, and PlanetScale - Vector Types and Indexes.
- PostgreSQL: A popular RDBMS with a strong community, offering pgvector, an open source vector similarity extension that delivers CREATE TABLE items, INSERT INTO items, and SELECT FROM items ORDER BY embedding.
- MongoDB: A popular NoSQL database with a strong developer community, a free community edition, and horizontal scaling, with vector search supported in MongoDB version 7.0.2 or 6.0.11 or later versions that avoids “synchronization tax.”
- Pinecone: Serverless vector database supported on AWS, Azure, and GCP with hybrid search, namespaces, and many integrations.
- Weaviate: Open source/cloud-based managed service/run as containers vector database supported on AWS, Azure, and GCP with hybrid search, vector index compression, and many integrations.
For an in-depth review of vector database and vector search for AI apps, including use cases, challenges, and more, you can view an archived version of the webinar here.