The dramatic advances in generational AI—chatGPT in particular—have motivated almost all technology companies to find an AI story that can break through the heavily AI-oriented technology news feeds.
It can be hard for database companies to tell an AI story. While most AI is data-driven and therefore dependent on database technology, advances in database technology have not in themselves driven advances in AI, and AI has generally not yet revolutionized database technology.
In that context, let’s look at the new MongoDB Vector search index capability. MongoDB says Vector Search “dramatically simplifies bringing generative AI and semantic search into applications for highly engaging end-user experiences.” It sounds great, but even after reading the details several times, I found it hard to precisely understand the technology or it’s applications.
After further investigation, I eventually concluded that several interesting MongoDB capabilities are involved here, each of which can be used to build solutions that might claim to involve artificial intelligence or at least, machine learning.
For instance, let’s say we are building a recommendation system for a streaming video service. We obviously want to be able to recommend movies “like” the one you have just watched. But how do we find these similar movies?
In machine learning, this sort of problem would generally be solved by a classification algorithm such as K-nearest neighbors (KNN). Using KNN, we would classify each movie based on relevant dimensions (or “vectors”). For instance, When Harry Met Sally might get 9/10 for Romance, 0/10 for Action and 5/10 for Comedy. KNN will find other movies that are “nearest” to When Harry Met Sally based on having the most similar rankings.
To start with, we create a trigger on our movies database that issues a call to an AI service (such as the OpenAI ChatGPT service) whenever a movie is inserted or modified. The AI service returns “vectors” representing ratings on various AI-created scales. We send the plot synopsis of our movies to OpenAI, and it returns an array of vectors that we can then store in a new attribute within the MongoDB collection.
We then create a vector index on this new attribute. The vector index is a new type of search index of type knnVector that can index the array of vectors.
Once the vector index is created, we can search for similar movies using an aggregation framework query. The aggregation framework $search command has a new option knnBeta, which performs a K-nearest neighbors analysis for you. You provide the vector entry for a movie (calling OpenAI if necessary to generate the vector), and the aggregation query will return the most “similar” movies.
Integrating MongoDB with an external AI service and performing KNN classification is exciting, even if it isn’t all that revolutionary. The KNN algorithm was first developed in 1951, and KNN has been routinely used in Web applications for several decades. However, until recently, database applications that want to use KNN have had to use awkward workarounds involving blobs or pre-computing the nearest neighbors. MongoDB vector search indexes provide a better solution for MongoDB applications that want to create recommendation and other classification systems.