Retrieval and Semantic Search

To have a model answer questions based on a knowledge base, the most direct idea is to stuff the entire knowledge base into the model and let it find answers in it by itself. This approach is not viable in the vast majority of real-world scenarios, for three reasons.

The first is capacity: the model's context window is finite (see its deep-dive page for details of this mechanism), whereas real knowledge bases often contain tens of thousands of documents, and their total size far exceeds what the window can hold and simply cannot fit. The second is cost and speed: even if it barely fits, taking in massive amounts of text as input makes inference slow and expensive, and token consumption grows rapidly with input length. The third is quality: mixing a large amount of question-irrelevant content into the input dilutes the signal, and the model may be overwhelmed by noise, exhibiting the "lost in the middle" effect—information located in the middle of a long context is more likely to be ignored by the model, so it fails to grasp the truly critical information.

Therefore the correct approach is the reverse: first pick out a small number of documents most relevant to the question from the vast collection, feed only that small set to the model, and let the model answer within that narrowed, clean scope. This "picking" process is retrieval.

In a nutshell, retrieval is the "finding a needle in a haystack" step—accurately pulling out a small amount of material relevant to the question from the knowledge base for the model to answer with. Whether it is pulled out accurately directly determines whether the answer is good: if key evidence is not retrieved, no matter how strong the model is, it cannot answer; if everything retrieved is noise, the model will likewise be led astray. Retrieval is therefore the front-line gate of the entire pipeline, and its quality sets an upper limit on the quality of the final answer.

Abstract retrieval as a component: its inputs are the user query, the accessible knowledge base, and the context budget; its output is a small number of candidate snippets and their sources. What it does is drastically narrow the scope of material before the model's limited window is filled, thereby reducing cost, latency, and noise at the same time. Note that a snippet being recalled only indicates that it is relevant to the query—it is a candidate "worth looking at"; it does not guarantee that it is sufficient to answer the question, nor that its content is correct—these judgments are left for the subsequent model reasoning to complete.

Retrieval is also not unconditionally necessary. When the knowledge base is small and can be completely and safely loaded into context, feeding it all in directly is a reasonable choice; in that case, the benefit of retrieval is offset by its own overhead. Therefore, whether to introduce retrieval depends on the relationship between knowledge base size and context budget: the closer the size is to the budget limit, the greater the value of retrieval.

The task of “finding relevant documents” can be approached from two fundamentally different angles: keyword retrieval and semantic retrieval.

Keyword retrieval matches on the literal level: whether the same words appear in the query and the document. Its implementation relies on traditional search techniques such as inverted indexes—maintaining a list of “which documents contain each word” in advance, and then directly looking up documents that contain those words at query time. Semantic retrieval matches on meaning: it embeds both the query and the document into vectors, and then finds the nearest neighbors to the query vector.

Each approach has its own blind spot. Keyword retrieval excels at exact words, proper nouns, and identifiers: querying an object with a unique literal form such as “Order Number 2026-0715” hits the mark immediately. But as soon as the user rephrases—for example, querying “return” while the document writes “return process,” the two share no common words, and keyword retrieval will simply miss this highly relevant document. Semantic retrieval fills exactly this gap: because “return” and “return process” are similar in meaning, their embedding vectors are close together, so it can recall the document. Conversely, semantic retrieval is not necessarily reliable for identifiers and proper nouns that are unique literally but semantically rare, so it excels at rephrasing, synonyms, and cross-language scenarios, rather than replacing literal matching.

Semantic retrieval works by turning each document and the query into vectors—similar meanings produce close vectors; see the “Embedding” deep-dive page for the mechanism—and then computing the cosine similarity between the query vector and each document vector, taking the top few with the highest similarity. Cosine similarity measures how close the directions of two vectors are:

cos(q, d) = q·d / (‖q‖ × ‖d‖)

That is, the dot product of query vector q and document vector d, divided by the product of their norms.

A toy two-dimensional vector calculation makes each step clear. Let query q = (1,1), fragment A = (1,0), fragment B = (2,2). For A: dot product q·A = 1×1 + 1×0 = 1, norm ‖q‖ = $\sqrt{\text{2}}$, ‖A‖ = 1, so cos(q, A) = 1/($\sqrt{\text{2}}$ × 1) = 1/$\sqrt{\text{2}}$ ≈ 0.707. For B: dot product q·B = 1×2 + 1×2 = 4, norm ‖B‖ = $\sqrt{\text{8}}$, so cos(q, B) = 4/($\sqrt{\text{2}}$ × $\sqrt{\text{8}}$) = 1. B's higher score means that in this representation space, B's direction is closer to the query. Real embeddings have hundreds to thousands of dimensions, much higher, but the logic is the same: the score only represents the geometric similarity learned by the model, not factual correctness, nor that it is sufficient to answer the question.

Since both retrieval methods have their blind spots, practice commonly uses “hybrid retrieval”: fusing the keyword and semantic results together is often more robust than using either alone—neither missing precise identifier terms nor rephrased expressions.

The following worked example uses the query “refund period” to show the scores from both paths and the fused ranking:

This table also illustrates that scores cannot be judged merely by “how high.” Both A and B can answer “how long can one apply”; C, although it overlaps heavily with the query literally (keyword score as high as 0.88), is about how long after approval the refund arrives, answering a different question. If we take Top-2 as A and B, both relevant fragments are recalled, Recall@2 = 2/2 = 100%; if we sort by keyword alone and take A and C, B is pushed out of the top two, Recall@2 = 1/2 = 50%. This comparison points to a more fundamental principle: retrieval evaluation requires first labeling “what counts as relevant,” not staring at similarity scores and feeling satisfied.

From a component perspective, hybrid retrieval takes as input the query q, document vector d, keyword score, and semantic score, and outputs a fused candidate ranking. Each branch has its applicable boundary: requirements involving precise identifiers should trust keywords first, while rephrased and synonymous expressions should trust semantics first; production environments usually fuse the two branches and then re-rank.

Using semantic search as an example, a single retrieval has two stages behind the scenes: offline index building and online retrieval.

Figure 1: Two stages of offline index building and online retrieval. The top half is the offline stage: raw documents → split into small chunks → embed each chunk → store in a vector database; the bottom half is the online stage: user query → embed → find the most similar Top-K chunks in the database (approximate nearest neighbor) → return candidates.

The offline stage is one-time preparation: split documents into small chunks, embed each chunk, and store them in a vector database (see the deep-dive pages “Document Chunking” and “Vector Databases” for details on splitting and storage). It turns “documents” into a “set of vectors that can be compared quickly”. Because this step happens before a query arrives, its cost does not enter the latency of a single Q&A, and the cost is shared across the entire database.

The online stage must be fast: after a user query arrives, first embed the query into a vector using the same rules, then find and return the Top-K chunks most similar to it in the database. This step uses approximate nearest neighbor (ANN) search—making exact comparisons across massive vectors is too expensive, so approximate algorithms use index structures to make a trade-off between quality and speed, quickly returning neighbors that are good enough.

There is a hard consistency constraint between the two stages: queries must be encoded with the same representation rules used when building the offline index. In other words, the model and configuration used to embed queries online must be the same as those used to embed documents offline; otherwise the vectors on the two sides lie in different geometric spaces, and the “near/far” comparison between them is meaningless. Therefore, once the embedding model is changed, the old index becomes invalid entirely; changes to chunking or permission scope likewise alter what the index should contain. After any of these three changes occurs, the index must be rebuilt or updated—an index is always just a snapshot under a given configuration.

Abstract the whole pipeline into components: the inputs are raw documents and online queries, and the output is Top-K candidates, each with a chunk ID, version, permissions, and similarity. The permissions component reminds us that retrieval is not a purely mathematical problem—it must operate within the scope of “what the user is authorized to see”, and retrieval results must not exceed that authorization.

Finally, a note on interpreting the returned order: the ordering of candidates is the retriever's judgment of “which is more relevant”, placing the most likely evidence first, but this is not the final ranking of facts; chunks ranked later are not necessarily less reliable, and the final choice is still made by the downstream reasoning stage.

The generative model in RAG is usually very strong and the output is fluent, but in engineering practice people repeatedly emphasize that “the problem is mostly in retrieval.” The reason has to do with the structure of RAG itself.

The logic of RAG is “retrieve first, then have the model answer based on the retrieved material”; retrieval is a precondition for generation. If the retrieval step does not fetch the correct documents, the model has no correct material at all—it either cannot answer or simply makes things up, i.e., hallucination. What is at work here is “garbage in, garbage out”: if the input material is wrong, no matter how strong the model is, it cannot recover, because the model can only look for evidence in the context it receives and cannot “recall” out of thin air content that never appeared in its input. Therefore, in many RAG systems, the performance ceiling is not the model but retrieval: no matter how high the generation ceiling is, it will be directly cut off by failures in the retrieval step.

Retrieval failures have several common concrete forms. First, the query and document wording are too far apart: the user uses a completely different expression from the knowledge base, and neither literal matching nor semantic similarity can push the correct document to the top. Second, chunking scatters key information: the evidence for an answer in a document is split across different chunks, no single chunk alone is enough to support an answer, and even if retrieval recalls one of them, it is of no help. Third, relevant documents are pushed out of the Top-K by more irrelevant documents: Top-K has limited capacity; when noisy documents have inflated scores, the truly relevant chunks fall outside K and never enter the context.

When diagnosing retrieval as the bottleneck, it can be abstracted into a decision component: input the target question, necessary evidence, and recall results, and output one of three states—“evidence recalled”, “pushed out by noise”, or “not in the knowledge base”. This three-way classification separates symptoms from causes: in the first state the bottleneck is not in retrieval, and the problem lies further downstream; the second indicates a problem with ranking or fusion strategy; the third means the knowledge base itself lacks that evidence, and no retrieval algorithm can conjure up content that does not exist.

Because the generative model can only use material that enters the context, when the necessary evidence is not recalled, no matter how strong the model is, it cannot reliably recover the answer. Conversely, adequate retrieval does not guarantee that assembly and generation are correct—after recalling the correct chunks, the model may still assemble them incorrectly, read them incorrectly, or reason incorrectly. Therefore, when locating a RAG failure, the first step is always to determine at which layer the evidence chain breaks: was it not retrieved, pushed out, or retrieved but not used correctly.

Building on the previous example: the initial Top-K includes documents such as 'Refund Crediting Time' that are lexically similar but do not answer the question. How can we improve recall without letting noise drown out the answer? There are four common improvement methods, each acting on a different stage of the retrieval pipeline.

Reranking is the most direct lever. First use fast, coarse retrieval to pull back a batch of candidates (say 50), then use a finer model to precisely compute relevance item by item, reorder them, and take the top few. Its division of labor is 'coarse filtering first, fine ranking later': coarse filtering is responsible for covering a wide range cheaply, and fine ranking is responsible for surfacing what is truly worth reading (see the 'Reranking' deep-dive page for details).

Query rewriting acts on the input side. User questions are often colloquial and imply context, so using them directly for retrieval is not effective; rewriting or expanding them into a more retrieval-friendly form, or splitting them into multiple subqueries that are retrieved separately and then merged, can significantly improve hit rate (this belongs to 'Advanced RAG' techniques).

Hybrid search fuses results from two paths—keyword and semantic—so that each branch compensates for the other's blind spots: exact identifiers rely on keywords, and paraphrases rely on semantics.

Chunk tuning acts during the offline indexing stage. If chunks are too large, irrelevant content gets mixed in as noise; if too small, complete meaning is cut apart. The chunking method directly affects whether queries can hit the correct passages and is a variable that needs repeated tuning depending on document type.

Each of these methods can improve retrieval, but how do we know whether retrieval is accurate? The answer can only be evaluation: we commonly look at metrics such as recall (whether what should have been retrieved was retrieved). Without evaluation, retrieval quality can only be judged by feel, and optimization has no place to start (see the 'LLM Application Evaluation' deep-dive page for methodology).

During evaluation, we must also hold to one principle: similarity does not equal answerability. A document may be topically relevant but not contain the object, time, or condition the question requires—just as 'Refund Arrival Time' and 'Refund Period' are highly similar literally, but the former does not answer 'how long one can apply'. Rerankers also make mistakes; they may likewise put topically relevant passages that lack key elements at the top. Therefore production evaluation should look at multiple dimensions simultaneously: relevance, answer coverage, timeliness, permission filtering, and whether the final citation truly supports the conclusion.

Abstract this whole set of improvements into components: the inputs are initial candidates, query, chunks, permission and timeliness information, and the output is a smaller candidate set after query rewriting, hybrid recall, and reranking. The division of labor at each stage can be summarized as: coarse recall aims to miss nothing, fine ranking aims to put passages that can truly answer the question at the front; the final delivered result must be able to simultaneously explain recall, noise, version, and permission, rather than only giving a ranked list. Two boundaries need to be remembered: an excessively large Top-K may aggravate the model's 'Lost in the Middle' effect—the longer the context, the easier it is for mid-position evidence to be ignored; the reranker itself must also be independently evaluated, and cannot be assumed to be more reliable than coarse ranking by default.

If both systems retrieve the correct document, why isn’t ranking it 1st as good as ranking it 20th? Because downstream can only consume a limited number of positions: only results within the Top-K can enter the context, and results ranked earlier are more likely to be actually used by the model. So evaluation metrics care not only about “whether it is there”, but also “at which position” and “whether the earlier content is correct”. Common metrics each answer a different question and each has a blind spot:

The formulas for the two core metrics are intuitive. MRR is the mean reciprocal rank: for each query, take the reciprocal of the position of the first relevant result, then average over all queries. Suppose in three test queries, the first relevant passage ranks 1st, 2nd, and 5th, respectively, then

MRR = (1 + 1/2 + 1/5)/3 ≈ 0.567

The meaning of each term is: ranking 1st contributes 1, ranking 2nd contributes 1/2, ranking 5th contributes only 1/5—the later the first evidence appears, the closer that query's contribution is to 0.

Recall@K is the hit rate: Recall@K = Hit@K / Relevant, where Hit@K is the number of relevant items hit in the Top-K results, and Relevant is the total number of relevant items for that query. It only asks “how much of what should be retrieved was retrieved”, completely ignoring where the retrieved items are ranked, and also ignoring how much irrelevant content is mixed into the Top-K.

The blind spots of these two metrics are complementary; only by looking at them together do we get the full picture. Consider a task where each query actually needs two passages—“general rule + exception”—to answer correctly, but the system always ranks the general rule 1st and misses the exception. In this case MRR still looks good—the first relevant result is 1st every time—but the exception is never recalled, so the answer is necessarily wrong. Therefore, looking only at MRR gives false reassurance; you must also look at Recall@K to discover the missing evidence.

A more general principle is: the metric is determined by the task's needs. Tasks that must not miss evidence prioritize Recall@K; tasks with tight context budgets and a need to control noise prioritize Precision@K; tasks where “finding the first answer entry point is enough” are suited to MRR; tasks with complete graded relevance annotations are suited to nDCG@K. Conversely, automatically treating a single nDCG on a leaderboard as product quality is a common misuse—it measures ranking quality, not task completion quality.

Offline relevance is not the final success either. During evaluation, you should also slice by document version, permissions, language, query length, and hard examples to see performance differences under different conditions; after launch, observe online behavioral signals such as citation support rate, handling of no-answer cases, user re-asking, and task completion. Pay special attention to click-through rate: it may favor clickbait and cannot be directly used as a correctness label.

Assemble these metrics into an evaluation component: input N queries, the annotated relevant set for each query, the ranking position rankᵢ of each relevant result, and truncation K; output Recall@K, Precision@K, MRR, and nDCG. Here MRR is the average over queries of the reciprocal rank of the first relevant result, and Recall@K equals Hit@K divided by Relevant. When interpreting results, keep one boundary in mind: a high MRR only means the first piece of evidence is near the top; it does not mean that both the general rule and the exception have been recalled. Metrics must be chosen according to the task's evidence needs, not by habit.

By stringing together the previous sections, you can see that every step in the entire design chain is derived from the previous constraint, with everything interlocking.

The starting point of the chain is a hard constraint: the context window cannot fit the entire knowledge base (§1). Since you cannot feed everything in, you must first retrieve the most relevant small amount of material and then hand it to the model. This constraint also determines the goal of retrieval—to use the smallest possible amount of material while covering the evidence necessary to answer.

Once you have the need to 'select', the next step is 'how to select'. Retrieval has two fundamentally different paths: keyword retrieval on the literal text and semantic retrieval on meaning—the latter relies on embeddings to turn queries and documents into vectors and then find nearest neighbors (§2). Each path has its own blind spots, so in practice they are often mixed: keyword retrieval preserves exact identifiers, while semantic retrieval preserves expressions that use different wording.

Then comes the engineering implementation. The entire pipeline is divided into two stages: offline index building (chunking → embedding → storing in the vector database) and online querying (query embedding → nearest neighbor search → Top-K) (§3). The offline stage amortizes costs, the online stage pursues low latency, and querying and index building must use the same representation rules—any configuration change requires rebuilding the index.

Why is it worth being rigorous about these steps? Because retrieval is the bottleneck of RAG (§4): if retrieval picks the wrong material, the model has no correct material in hand and can only answer incorrectly or fabricate. Retrieval quality sets a ceiling on the overall system's performance—no matter how strong the later model is, it cannot exceed it.

To address this bottleneck, improvement measures fall into four parts: reranking refines the coarse candidates, query rewriting makes colloquial questions more suitable for retrieval, hybrid retrieval fuses the two signal paths, and adjusting chunking changes the granularity of matches; whether all of these measures are effective can only be measured through evaluation (§5).

Evaluation metrics themselves are also divided along the causal chain (§6): Recall@K handles missed evidence—whether the evidence that should be in the Top-K is included; Precision@K handles noise—how much irrelevant content is mixed into the Top-K; MRR and nDCG handle ranking—whether the first relevant item comes early and whether multi-level relevance is ordered correctly. But offline metrics are not the endpoint: ultimately you still need to connect retrieval results to whether citations support conclusions and whether user tasks succeed. If retrieval metrics look great, but citations do not support the answer, or users keep re-asking, it means the chain broke at a step after retrieval.

The entire chain can be self-checked with one question: why can semantic retrieval recall documents that share no common keywords? Because documents and queries are embedded into the same geometric space—when meanings are close, vectors are close—so nearest neighbor search can cross the gap of literal wording. And why is retrieval the ceiling of RAG effectiveness? Because the model can only make use of material that enters the context; evidence that was not retrieved cannot be restored out of thin air. If you can clearly explain these two 'whys', you have grasped the core of retrieval.

Retrieval is not an island; it sits in the middle of a clear chain of conceptual dependencies. Before understanding this page, you need several prerequisite concepts; after mastering this page, there are several immediate extensions and further directions to continue exploring.

The prerequisite layer lays the two foundations of this page. Embeddings are the mathematical basis of semantic search—without the step of turning text into vectors, there is no comparison where similar meanings have similar vectors; Context Window and Tokens and Tokenization explain why retrieval must exist: the window is limited and billing is by token, so you need to narrow the material scope first.

The five core concepts on this page form a complete mechanism chain: keyword and semantic search are two complementary matching routes; nearest neighbor is how the semantic route searches in vector space; offline/online is the engineering division into two stages; retrieval as bottleneck explains why it is worth investing in; and reranking is the refinement step after coarse recall.

Immediate extensions unfold outward along this chain: Retrieval-Augmented Generation (RAG) is the most direct scenario served by retrieval; Vector Databases are the storage substrate for offline index construction; document splitting determines the hit granularity of the offline stage; reranking deserves a separate page for in-depth reading; Advanced RAG covers more complex orchestration such as Query Rewriting; and evaluation is the only way to verify whether all of this is truly effective.

Further directions widen the perspective: Knowledge Graphs and GraphRAG extend retrieval from vector similarity to structured relationships; Lost in the Middle explains why retrieving more is not necessarily better; and Context Engineering treats “what material to give the model” itself as a system problem to design. Learning along this dependency chain allows each layer to directly answer the questions left by the previous layer.