Advanced RAG: Making Retrieval a Diagnosable, Iterative Evidence Process

When the answer of a Retrieval-Augmented Generation (RAG) system is wrong, the most common impulse is to immediately add components: add a reranker, change the embedding model, enable multi-query. But the error can occur at any layer of the entire pipeline—parsing, chunking, query construction, retrieval, reranking, context utilization, or generation—and the new component is likely added at a position unrelated to the failure. Therefore, first answer the question “which layer deviated from expectations”, then decide what to change.

The inputs to layered diagnosis are the artifacts of each layer in the pipeline: the original query, parsed document chunks, retrieved candidates, reranked results, assembled final context, and final answer; the output is the first layer that deviated from expectations, and a falsifiable repair hypothesis. Saving the artifacts of each layer and annotating them with evidence is the prerequisite for locating the bottleneck and the line of defense against blindly stacking components. When the candidate set does not contain the correct information at all, the problem lies in the query or retrieval; when the candidates cover the correct information but the answer is still wrong, the problem lies in assembly or generation; when citations conflict with each other, the problem often lies in old or incorrect versions mixed into retrieval.

The falsifiable hypothesis is the core tool of this layered diagnosis. Saying vaguely that “retrieval quality is poor” cannot guide changes; each failure should be written as a specific assertion that can be overturned by experiment, for example “if quality exceptions are added to candidates, the answer will be correct” and “if old versions are filtered out, citation conflicts will disappear.” Then change only the layer the hypothesis points to, and rerun on the same query set. The same data and a single variable are the prerequisites for subsequent conclusions to hold: if multiple links are changed at once, gains cannot be attributed and failures cannot be located.

There are two typical interpretations of rerun results. If candidate coverage increases while the answer remains unchanged, the bottleneck has shifted to assembly or generation, retrieval is no longer the main contradiction, and continuing to invest in retrieval is wasteful; if the answer improves but latency exceeds budget, there is no need to make all queries bear the cost of the complex pipeline—route the complex pipeline only to difficult queries. Diagnosis is therefore not only about finding the point of error, but also about deciding where to continue investing and where to stop.

This explains why components cannot be directly stacked before the failure is located. Layered experiments—propose a hypothesis, change only one layer, rerun the same query set—better clarify where the gains come from than simultaneously enabling multi-query, HyDE, a reranker, and an AI Agent, and they preserve a safe fallback path: each layer has a clear before-and-after comparison, and rolling back any layer will not affect the other layers.

The questions users ask are often not the best search terms: natural language mixes ambiguity, omitted entities, dates, and negation conditions, and the retrieval system may not be able to directly read from a single sentence what to look for. Query rewriting targets this gap. Common techniques include disambiguation, adding entities, generating multiple queries, and constructing hypothetical documents with HyDE. The common effect of these techniques is to complete the search expression and improve recall, but they may also introduce incorrect assumptions, so both the original question and the rewritten result should be retained and auditable.

The input to the rewriter is the original question and the entities, dates, negations, and ambiguities that can be identified in it; the output is a structured rewrite or multiple subqueries that preserve these constraints. The key is that rewriting only supplements the search expression without replacing user intent. Therefore, the rewriter should output structured differences: which entity was added, which subquestions were split off, which dates and negation conditions were retained. If the differences are not written out, later reviewers cannot tell whether a retrieval shift was a deliberate expansion or an accident.

The cost of dropping negation can be seen clearly with an example: after “Can’t I get a refund?” is changed to “refund process”, the retrieval results are highly similar in topic, but the eligibility judgment has been lost—the user is asking “Do I have refund eligibility?”, while the rewritten query looks for “how to go through the refund process”. Topic similarity masks the semantic deviation; even high recall is collecting evidence along the wrong premise. Similarly, newly added entities can also lead retrieval to the wrong premise, for example, binding a question to some unrelated product line.

Therefore, rewriting and acceptance must be separated. The sole purpose of rewriting is to find evidence; the final generation must be accepted using the original question: whether the answer still answers what the user originally asked. HyDE also follows this rule—hypothetical documents are only a means of constructing a search representation. Using a generated “ideal answer” to match documents does not mean that this hypothetical content itself has binding force. Only by retaining the original question, retaining the rewrite differences, and retaining the sources can every expansion step be traceable and reversible.

Dense vector retrieval and keyword retrieval each have a region the other cannot reach: keyword retrieval excels at exact matching of names, model numbers, and digits, but is powerless against synonym rewriting; dense vectors excel at semantic-level rewrite matching, but tend to miss proper nouns and identifiers that must be matched verbatim. Hybrid recall runs two types of retrievers in parallel on the same query, then fuses the results into a candidate list, trading complementary coverage for recall rate.

The difficulty of fusion lies in the fact that the scores of the two types of retrievers cannot be directly compared—BM25 scores and cosine similarity are on completely different scales. Reciprocal Rank Fusion (RRF) bypasses scores and uses only ranks. It takes as input the candidate ranks rⱼ(d) given by each of the J retrieval paths and a smoothing constant k, and outputs a fused score for each document d:

RRF(d) = Σⱼ 1/(k + rⱼ(d))

Here j is the retrieval path index, and rⱼ(d) is the rank of document d in the j-th retrieval path. Each path contributes a term according to 1/(k + rⱼ(d)): the higher a document is ranked, the smaller rⱼ(d), so the larger its contribution; documents ranked high in multiple paths accumulate a higher total score. k is a smoothing constant that prevents the top-ranked item from being too large and dominating the fusion, and also flattens the gap between adjacent ranks, reducing the impact of minor fluctuations in a single path on the fusion result. Fusing by rank rather than summing scores fundamentally avoids the problem of directly summing quantities with different scales.

Fusion yields a re-ranked candidate set, and that is all. It does not judge the timeliness, permissions, or authenticity of documents; these attributes need to be handled separately in downstream stages of the pipeline. How the weights are allocated should also be verified by query type: which query types rely mainly on keywords and which on semantics needs actual validation, not guesswork. At the same time, the original ranks output by each retrieval path should be preserved—the fused score cannot explain a particular regression; only by tracing back the original rank each path assigned to a relevant document can we identify which retrieval path went wrong.

Although the fused candidates are all relevant to the query, they may still be duplicates of one another, conflict with each other, or be only marginally related. What reranking and context assembly address is “what to do after relevance”: turning candidates into a context that a generative model can reliably use.

Reranking is usually delegated to a cross-encoder. It feeds the query and document in pairs into the same model for joint encoding, judging relevance through token-by-token interaction, and is therefore more accurate than a bi-encoder that encodes each separately and then computes similarity; the downside is much greater computational load, so it can only be used on a small set after the number of candidates has been reduced. The scores output by fine ranking indicate “whether this passage is suitable for answering,” not “whether this passage is trustworthy”—being top-ranked and relevant only indicates suitability for answering, not trustworthiness.

The inputs to the assembly stage are not just the reranking scores; they also include each candidate's source, adjacent chunks, version, and permissions. The output is a deduplicated context that preserves necessary conflicts and citations. Deduplicating by source can merge duplicate chunks from the same document that have been split apart, and expand adjacent chunks to restore fragments into complete paragraphs, preventing the generative model from seeing only half a sentence. At the same time, filter out expired content and content without access permission. Different viewpoints and temporal versions cannot simply be merged: conflicting authoritative evidence must be explicitly retained, and the conflict should be explained during generation, rather than silently discarding one side. Key evidence should be placed near the task-relevant location, with citations, so that every claim in the final answer can be traced back to a specific source in the context.

Some questions cannot be answered with a single retrieval: you must first find A before you know to look for B. For example, if you want to know “which university the scholar who proposed a certain theorem teaches at,” you first need to find the scholar associated with the theorem, then use the scholar’s name to search for their affiliated institution. Multi-hop retrieval breaks such questions into sub-questions, retrieves step by step, and records the intermediate entities discovered at each step into evidence slots; when the existing evidence is insufficient to answer, or when the evidence conflicts with each other, it triggers supplementary retrieval to continue.

The input of a multi-hop loop is the original question, the already filled known evidence slots, and the current gap; the output is the next sub-query, the newly discovered intermediate entities, and the updated slot state. Each hop has reason to continue only when it fills a gap or resolves a conflict; if the current evidence is already sufficient, continuing retrieval only adds noise. Each step must also verify entities and sources, because erroneous entities cause subsequent retrieval to snowball: once the first hop writes the wrong intermediate entity into the slot, all subsequent queries will revolve around this incorrect premise, drifting further with each search, and later correct results also cannot easily undo the earlier deviation.

Therefore, multi-hop retrieval must include hard boundaries: a maximum number of steps, a retrieval budget, and termination on no progress. Stop when the step limit is reached or the budget is exhausted, answer with the available evidence and explain the gap; if several consecutive hops fill no slots, it should likewise terminate. The value of multi-hop lies in turning retrieval into a verifiable, explainable evidence chain, rather than an endlessly extending search loop.

Advanced pipelines cannot prove themselves by 'feeling stronger.' The inputs to evaluation are a fixed query set, and comparisons between basic Retrieval-Augmented Generation (RAG) and each single-component variant; the outputs are differences in Recall, NDCG, evidence faithfulness, answer success, latency, and cost. A fixed query set ensures every change is measured on the same scale, and single-component variants constitute component ablation: each time only add or remove one component to see exactly which metric it improves.

Component ablation answers end-to-end questions, not local questions. A component improving its own local metrics—such as higher recall or more accurate ranking—does not automatically mean the system is better. If reranking improves ranking quality but answer success does not change, it indicates the bottleneck is already in a later stage; at this point, you should examine context utilization instead of continuing to add components to the front end. Metrics must be tied to real outcomes: ranking and recall are intermediate metrics, while answer success and evidence faithfulness are the task itself; latency and cost are the price any improvement must pay, and all four must be reported together.

"More retrieval does not equal more truth" is the most easily overlooked boundary in upgrade decisions. Expanding recall often brings duplicate chunks, outdated content, and low-credibility sources into the context together; noise is amplified as the number of candidates increases, thereby diluting the originally reliable evidence. Therefore, components without a stable net benefit should be removed: if ablation shows that a component has no repeatable positive contribution to answer, faithfulness, latency, and cost, it is not worth keeping in the pipeline.

“Can a quality problem 35 days after signing for delivery be refunded?” is not a query that can be answered simply by increasing top-k. It simultaneously involves general deadlines, quality exceptions, time and eligibility judgments; any single-path retrieval will miss half of the evidence. The Advanced RAG process starts from the original question, first performs query decomposition, then runs dual-path retrieval with BM25 and vectors, and after fusion and reranking goes through timeliness and permission checks, ultimately handing over only materials that pass governance to generation. The output of each step can be inspected independently, and problems found are routed back to the corresponding layer for repair—it is not a component pipeline that can only move forward.

The rankings from dual-path retrieval and the fusion results can be fully recalculated. The table below lists, for candidates A through D, the BM25 rank, vector rank, and RRF score (k=60):

RRF uses only ranks, avoiding direct addition of incomparable BM25 and cosine scores; but the scores for A, C, and D in the table are almost identical, indicating that fusion is not the final arbiter—it is only responsible for expanding the candidate pool, not for judging who actually answers the question. Reranking must make fine-grained relevance judgments around “refund eligibility,” and timeliness filtering must also remove the seemingly not-low-relevance outdated policy D. Only by retaining A, B, and the order date in the final context can the conclusion be drawn: the general deadline has passed, but the quality exception may still apply.

This conclusion is reproducible. If only single-path top-2 were used, BM25 would give A+C, missing the quality exception; vectors would give B+D, missing the currently effective general rule. Fusion combines the evidence coverage of both paths, and reranking and timeliness rules then narrow the four candidates down to A+B, which together support the conclusion. The final answer can only cite materials that were actually retrieved and passed governance checks; any assertion not included in this set of evidence has no right to appear in the answer.

When the system continues searching after discovering insufficient evidence, the greatest danger is not spending a few more retrieval attempts, but turning a single missing-evidence case into an endless loop. The first step to prevent losing control is to declare the still-missing evidence slots in each retrieval round, for example, "order facts known, general rules known, quality exception unknown", rather than vaguely asking to "find more". With the slot list, a new retrieval round counts as making progress only when it fills a slot, resolves a version conflict, or raises the source level; all other actions are idle.

Apart from idling, wrong repair directions are also common; common situations and corresponding actions are as follows:

Each row corresponds to the same principle: the fix should occur at the layer where the failure actually happens, not by making more forceful efforts elsewhere. When evidence conflicts, a majority of fragments are not necessarily correct; the right approach is to check dates, permissions, and authoritative sources.

Stop conditions must be as hard as progress conditions. Preset maximum hop count, maximum query count, token or latency budget, and no-progress termination should be set in advance; after reaching any boundary, the output is divided into three parts: known, unknown, and items requiring human confirmation. Stop after two consecutive rounds with no new evidence slots and transparently report the insufficiency, instead of letting the Agent freely rewrite queries to continue looping.

There is another failure boundary belonging to the evidence source itself: the "hypothetical document" generated by HyDE is only used to construct a search representation, not evidence. If the hypothetical text fabricates product names or clauses, subsequent retrieval may snowball around erroneous premises. Likewise, content written by the Agent itself cannot be elevated to evidence for the answer. Answers can only cite materials that were actually retrieved and have passed version and permission checks.

All the techniques of Advanced RAG ultimately reduce to the same causal chain from problem to practice. The starting point of the chain is recording the real failures of basic RAG, not a preconceived list of solutions. Once you have failure samples, locate which layer they occur in—query, retrieval, ranking, or generation—and then add components only for the targeted failures you located, one change per failure. At each step, preserve evidence and sources so that the output of any layer can be re-examined and rolled back. Multi-hop and error-correction loops must set boundaries and handle conflicts explicitly, preventing retrieval from extending indefinitely on incorrect premises. Finally, use a fixed query set to perform component ablation, verifying that the added stage produces end-to-end gains rather than just improvements in a local metric that give false comfort. Each step forward in the chain answers the same question: which failure of which type of query disappeared because of this change, and at what cost.