How Semantic Search Works

The Problem with Keyword Search

Traditional keyword search, powered by algorithms like BM25 and TF-IDF, matches documents to queries based on shared terms. The query "automobile maintenance" matches documents containing those exact words but misses documents about "car repair," "vehicle servicing," or "keeping your ride running." This vocabulary mismatch problem means that a perfectly relevant document can rank poorly, or not appear at all, simply because the author used different words than the searcher. Studies have estimated that vocabulary mismatch affects 30% to 40% of search queries, making it one of the most significant limitations of traditional search.

Keyword search also struggles with intent. The query "apple" could refer to the fruit, the technology company, or a record label. Keyword search has no way to disambiguate without additional context. Similarly, "jaguar speed" could be asking about the animal, the car, or the Atari game. Semantic search models trained on diverse text learn contextual representations that, when combined with user history and session context, can resolve these ambiguities more effectively than keyword matching alone.

Despite these limitations, keyword search has virtues that semantic search lacks. It is fast, deterministic, interpretable, and excellent at exact matching. A search for error code "ERR_CONNECTION_REFUSED" should return pages containing that exact string, not semantically similar pages about network connectivity problems. The best modern search systems combine both approaches, using keyword matching for precise queries and semantic matching for conceptual queries.

How Dense Retrieval Works

Dense retrieval, the technical foundation of semantic search, encodes text as dense vectors in a continuous embedding space. A neural encoder model (typically a transformer like BERT, fine-tuned for retrieval) processes a piece of text and produces a single vector, usually 384 to 1,024 dimensions, that represents its meaning. Two texts with similar meanings produce vectors that are close together (high cosine similarity), while texts with different meanings produce distant vectors.

The encoding process works in two phases. In the offline indexing phase, every document (or passage) in the collection is encoded into a vector and stored in a vector index. For a knowledge base with 10 million passages, this produces 10 million vectors. In the online query phase, the user's query is encoded into a vector using the same encoder, and the system performs a nearest-neighbor search to find the document vectors closest to the query vector. The top-k closest documents are returned as search results.

The mathematical operation at the core of dense retrieval is similarity computation. Cosine similarity measures the angle between two vectors: vectors pointing in the same direction have similarity 1, perpendicular vectors have similarity 0, and opposing vectors have similarity -1. Dot product is a simpler alternative that combines direction and magnitude. In practice, cosine similarity and normalized dot product produce identical rankings and are used interchangeably. The query vector is compared against every document vector (or, more efficiently, against a pre-filtered subset), and documents are ranked by decreasing similarity.

Embedding Models for Search

Bi-Encoder Architecture

The standard architecture for semantic search uses a bi-encoder: two copies of the same transformer model that independently encode the query and each document. The query encoder processes "how do I reset my password" and produces a query vector. The document encoder processes each passage in the knowledge base and produces a document vector. Similarity is computed as the dot product or cosine similarity between these independently computed vectors. The critical advantage of this architecture is that document vectors can be pre-computed and cached. Only the query needs to be encoded at search time, which takes milliseconds on modern hardware.

Bi-encoders are trained on pairs of queries and relevant documents. The training objective pushes the query vector and relevant document vector closer together while pushing the query vector and irrelevant document vectors apart. Contrastive learning with in-batch negatives is the standard training approach: in a batch of query-document pairs, each query's non-matching documents serve as negative examples. Hard negative mining improves training by specifically selecting difficult negative examples, documents that are topically related but do not actually answer the query, forcing the model to make finer distinctions.

Cross-Encoder Architecture

A cross-encoder processes the query and document together as a single input, concatenated with a separator token: "[query] [SEP] [document]." The transformer's self-attention allows every token in the query to attend directly to every token in the document, enabling much richer interaction modeling than the independent encoding of a bi-encoder. Cross-encoders consistently outperform bi-encoders on relevance ranking benchmarks by 5 to 15 percentage points.

The limitation of cross-encoders is computational cost. Because the query and document must be processed together, document representations cannot be pre-computed. Ranking 10 million documents with a cross-encoder would require 10 million forward passes through the transformer for every query, which is prohibitively expensive. The practical solution is a two-stage pipeline: a bi-encoder retrieves the top 100 to 1,000 candidates quickly, then a cross-encoder re-ranks those candidates for maximum accuracy. This retrieval-then-reranking architecture combines the speed of bi-encoders with the accuracy of cross-encoders.

Approximate Nearest Neighbor Search

Exact nearest-neighbor search compares the query vector against every document vector, which is linear in the collection size. For a collection of 100 million vectors, this means 100 million similarity computations per query, taking seconds even on fast hardware. Approximate nearest neighbor (ANN) algorithms trade a small amount of accuracy for orders-of-magnitude speed improvement, finding results that are very close to the true nearest neighbors in milliseconds.

HNSW (Hierarchical Navigable Small World) graphs are the most widely used ANN algorithm. They build a graph structure where each vector is a node connected to its nearest neighbors. Search begins at a random entry point and greedily navigates toward the query vector, following edges to progressively closer neighbors. Multiple hierarchical layers, where higher layers connect more distant nodes, allow the algorithm to quickly traverse large distances before fine-tuning the search at lower layers. HNSW typically finds 95%+ of the true nearest neighbors while examining fewer than 1% of the vectors in the collection.

IVF (Inverted File Index) partitions the vector space into clusters using k-means. At query time, only the clusters closest to the query are searched, dramatically reducing the number of comparisons. Product quantization (PQ) compresses vectors from hundreds of floating-point numbers to a few dozen bytes, reducing memory requirements by 10x to 50x while preserving enough distance information for approximate search. Libraries like FAISS (Facebook), ScaNN (Google), and Annoy (Spotify) implement these algorithms and are used in production at billions-of-queries-per-day scale.

Hybrid Search: Combining Keywords and Semantics

The most effective modern search systems combine keyword and semantic matching. Keyword search handles exact matching (product codes, error messages, proper nouns) and precision-critical queries where the user knows exactly what they are looking for. Semantic search handles conceptual queries, paraphrased questions, and situations where the user describes what they need rather than using the document's exact terminology. Combining both approaches covers the weaknesses of each.

Reciprocal Rank Fusion (RRF) is a simple, effective method for combining results from multiple retrieval systems. Each system produces a ranked list of results. RRF assigns each result a score based on its rank in each list (score = 1 / (k + rank), where k is a constant, typically 60), then sums scores across lists and re-ranks by total score. A document ranked #2 by keyword search and #5 by semantic search receives a higher combined score than a document ranked #1 by only one system. RRF is robust, requires no tuning, and consistently improves over either individual system.

Learned sparse representations, like SPLADE, bridge the gap between keyword and semantic approaches. SPLADE expands the query with semantically related terms using a learned model, so a search for "automobile maintenance" automatically includes expanded terms like "car," "vehicle," "repair," and "servicing." The expanded query is then matched using standard inverted index infrastructure (BM25), combining the conceptual understanding of neural models with the speed and interpretability of keyword search. This approach performs competitively with dense retrieval while running on existing search infrastructure.

Applications of Semantic Search

Enterprise knowledge management is one of the highest-value applications. Employees searching an internal wiki for "how to expense a conference registration" can find the relevant policy document even if it is titled "Travel and Professional Development Reimbursement Procedures" and never uses the word "expense." Semantic search makes organizational knowledge accessible through natural questions rather than requiring users to guess the exact terminology used in internal documentation.

E-commerce search uses semantic matching to understand shopping intent. A search for "something warm to wear running in winter" should return insulated running jackets, thermal leggings, and cold-weather running gear, none of which necessarily contain the search terms. Product discovery, where users browse without a specific product in mind, benefits enormously from semantic understanding of queries like "gift for a gardener who has everything" or "tools for someone learning woodworking."

Retrieval-augmented generation (RAG) relies on semantic search as its retrieval component. When a chatbot receives a question, it uses semantic search to find the most relevant passages from its knowledge base, includes those passages in the language model's context, and generates an answer grounded in the retrieved evidence. The quality of the semantic search directly determines the quality of the generated answer: if the retriever finds the wrong passages, the generator produces an incorrect or irrelevant response regardless of how capable the language model is.

Legal and patent search uses semantic matching to find prior art, related case law, and relevant statutes based on the meaning of a legal argument rather than specific citation numbers or legal jargon. Medical literature search helps clinicians find relevant research using clinical descriptions rather than MeSH headings or exact paper titles. Academic search engines like Semantic Scholar use semantic matching to recommend related papers and identify connections between research areas that keyword matching would miss.