Skip to main content
Profile picture
SourceTree logo with a blue background.

Semantic Search

AI / Agents / LLMs / Semantic Search • Embeddings

Semantic search is a technique that improves search accuracy by understanding the intent and contextual meaning of search queries. Unlike traditional keyword-based search methods, semantic search focuses on the relationships between words and concepts, enabling it to retrieve results that are more relevant to the user’s intent.

Semantic search enables finding relevant information based on the meaning and context of the query rather than just keyword matching. This is achieved by creating embeddings for text, images, or videos and storing them in vector databases, allowing for efficient similarity searches.

How does Semantic Search Work?

It is typically implemented using by generating embeddings for the data to be searched and the search queries. Embeddings are high-dimensional vector representations that capture the semantic meaning of the content. When a user submits a query, it is also converted into an embedding. The system then compares the query embedding with the stored embeddings in the vector database to find the most similar items based on distance metrics like cosine similarity or Euclidean distance.

The power of this approach is that embeddings can be created from any type of data, including text, images, and videos, making semantic search a versatile solution for various applications.

Vector Databases

Vector databases are specialized databases designed to store and manage high-dimensional vector data, such as embeddings. They provide efficient indexing and querying capabilities to perform similarity searches on large datasets. Popular vector databases include Pinecone, Milvus, ElasticSearch, Weaviate, These databases optimize the storage and retrieval of vector data, enabling fast and scalable semantic search operations.

Hybrid search combines traditional keyword-based search with semantic search techniques. It can also include specialized types of search, such as geospatial search. This approach leverages the strengths of both methods to provide more comprehensive search results. In a hybrid search system, the initial search may use keyword matching to filter results, followed by semantic search to rank and refine those results based on their relevance to the user’s intent. This combination enhances the overall search experience by ensuring that users receive results that are both contextually relevant and aligned with their specific queries.

An example of a hybrid search implementation could involve using ElasticSearch to perform keyword searches while also integrating vector search capabilities to rank results based on semantic similarity. For example a query for user could combine the following criteria:

  • Keyword Match for persons: named “John”
  • Geographic Location: geo search within 10 miles of a lat/lon coordinates or from a specified city, i.e. “New York”
  • Semantic Similarity:
    • documents related to “software engineering” or “machine learning” using vector search through embeddings
    • profile image similarity to a provided image of a person, again using image embeddings generated from a pre-trained model
  • Demographic Filters: age between 25-35, sex: male or female

Some databases, such as ElasticSearch and Weaviate, provide built-in support for hybrid search capabilities, allowing developers to easily implement this approach in their applications.

Creating Embeddings

semantic-search-image-2.png

Embeddings can be created using various pre-trained models or custom-trained models depending on the type of data. For text data, models like BERT, GPT, or Sentence Transformers can be used to generate meaningful embeddings. For images, models like CLIP or ResNet can be employed to create image embeddings. Videos can be processed by extracting key frames and generating embeddings for those frames using image models or by using specialized video embedding models.

There are many models available from major AI model providers such as OpenAI, Google, as well as open sourced models on HuggingFace, etc.

These models can be accessed via APIs or hosted within your application depending on the use case and requirements.

  • Recommendation Systems: Suggesting items based on user preferences and behavior.
  • E-commerce: Improving product search and recommendations based on user intent.
  • Image Search: Retrieving images similar to a given image or description.
  • Video Search: Locating videos based on content similarity.
  • Customer Support: Enhancing support systems by retrieving relevant knowledge base articles quickly and accurately.
  • Content Management: Organizing and retrieving documents based on semantic relevance rather simple keyword matching.