Lost in the Middle: When context fits, why might the model still fail to find it?

The “Lost in the Middle” phenomenon refers to an empirical observation: the same piece of information that is crucial for answering a question, simply because it is placed in the middle of a long context, becomes noticeably harder for the model to find and use, and performance drops accordingly. Intuitively, whatever content is placed in the context should be treated equally by the model; but experimental observations show otherwise—when relevant information is at different positions, the model's output quality changes in a measurable and broadly consistent direction, with the middle often being the weakest region.

This effect is not a conclusion based on intuition but a position effect that needs to be confirmed experimentally. To confirm it, the experimental inputs should be a set of inputs that differ from each other only in the position of the evidence: the same question, the same key evidence, the same batch of distractor documents, with the only change being where the key evidence is inserted in the context. The experimental outputs are metrics aggregated by position buckets, such as answer accuracy, or the citation hit rate for the correct document when the model generates an answer. The logic for determination is: if merely moving the position of the key evidence is sufficient to cause stable metric changes, then the conclusion that the system is position-sensitive is supported; conversely, if metric fluctuations across positions are non-directional within the margin of error, one cannot claim that Lost in the Middle exists.

The U-shaped curve frequently mentioned in such experiments also needs to be understood precisely. U-shaped refers to the fact that in a particular experiment, performance at the beginning and end positions of the context is better than at the middle position, so the curve plotted by position is high at both ends and low in the middle. It describes the observational results of that particular experiment and does not mean that every model, every context length, and every type of task will exhibit exactly the same curve shape.

The classic experimental approach is to place the same key evidence at the beginning, middle, or end of the context, while keeping other conditions as unchanged as possible. Under such controlled experiments, some models exhibit a U-shaped positional curve: content at the beginning and end is utilized better, while content in the middle is relatively weaker. It is worth noting that this is always an empirical phenomenon: it comes from observations of specific models, specific lengths, and specific tasks, and is not a universal law that necessarily holds for all models, all tasks, and all lengths. The fact that a particular model exhibits Lost in the Middle on a particular task does not mean that another model or another task will necessarily do so; conversely, the absence of the effect in some combination does not negate its existence in other combinations.

To illustrate with a concrete example: suppose there are 30 candidate documents, only one of which contains the answer. When feeding these 30 documents to the model, the position of the document containing the answer can be controlled manually. If merely changing the position of this document causes the answer accuracy to change accordingly, it indicates that the system is position-sensitive, and the Lost in the Middle effect holds for this system. The key in this example is the word “only”: keeping everything else unchanged, position is the only variable, so that the performance difference can be cleanly attributed to position rather than to the content itself.

Context window and effective context describe two different things. A context window is a capacity concept: it specifies the maximum number of tokens a model can receive in a single request; beyond that, the API typically rejects or truncates directly. It answers the question of “whether it can be put in,” but does not guarantee that every token put in will be used stably and equivalently. Effective context, on the other hand, is a performance concept: it refers to the extent to which and how well the model can actually use the information within it, given that it fits within the window. It answers the question of “whether it can be used after being put in.”

The difference in how the two are measured stems from the difference between input and output. To measure window capacity, the input is only the token count of the request, and the output is a single binary result: whether the API accepts or rejects. This boundary is determined by the upper limit set on the server side and is an engineering constraint. In contrast, to measure effective utilization, the input is much richer: in addition to the content itself, it includes where the key evidence is located, how many distractors are in the context, and what task is to be completed; the output is no longer just accepted or rejected, but task metrics such as accuracy and citation hit rate for the correct evidence. These metrics cannot be obtained directly from any API; they must be measured through experiments designed for the specific task.

Therefore, a statement such as “supports 128K” can only be interpreted as “can receive a capacity of 128K,” not as “can reliably retrieve evidence at any position within 128K.” The former is a capacity claim, the latter is task performance, and between them lies the model's actual ability to use the context. Treating a capacity claim directly as a guarantee of evidence-retrieval ability will lead to overestimating the system's reliability in long contexts. This is one of the reasons why many long-context products underperform expectations after launch.

Effective context is also affected by many factors: the difficulty of the task itself, the number and similarity of distractors, the task's requirements for retrieval ability, the structure of the prompt, and the model version. Under the same window specification, the effective range for a simple task may be much larger than for a difficult task; when distractors are dense, content in the middle positions is more easily submerged; even if different model versions have the same window number, their actual utilization ability can differ significantly. In other words, effective context is not a constant determined by hardware specifications, but the result of the combined effects of model capability, task characteristics, and input structure.

So, the window numbers provided by vendors cannot replace your own evaluation of position and length on real workloads. To determine whether a system can handle a particular long-context task, the correct approach is to test with your own task, your own distractors, and your own evidence distribution, rather than making inferences based on the window upper limit on the marketing page.

To measure the position effect as a curve, there is only one core approach: fix the evidence and the question, change only the relative position of the evidence in the context, and repeat with enough samples. Only position is the variable; keep everything else as constant as possible, so that observed performance differences can be attributed to position itself.

The inputs to a curve experiment are multiple pairable question instances, along with the corresponding positions, context lengths, and distractor settings; the outputs are accuracy, confidence intervals, and sample size under each condition. All three outputs are essential: accuracy gives a point estimate, confidence intervals give the range of variation, and sample size determines how credible that range is. For each question, generate multiple versions with the evidence at the beginning, middle, and end, and randomize the order of the remaining materials so that the question itself serves as its own control—the performance difference for the same question across three positions is much cleaner than performance differences between different questions, because question difficulty and required knowledge are controlled through pairing. If the differences on the curve are smaller than the statistical fluctuation range, you cannot claim a position effect; a curve that appears lower in the middle and higher at both ends is convincing only when the sample size is sufficient and the confidence intervals do not overlap with adjacent positions.

The relationship between accuracy and the variables can be written as:

A = f(p, L, d)

Here A is the task accuracy measured under the current conditions; p is the normalized relative position of the evidence in the context, with values close to 0 indicating near the beginning and close to 1 indicating near the end; L is the total token length of the context; d is the number or strength of similar distractors. This notation expresses that accuracy may vary jointly with the three variables, rather than presupposing that a fixed algebraic formula exists among them—how p, L, and d affect A is exactly what the experiment is designed to answer, so function notation f is used instead of a concrete expression.

When measuring, normalize p to the range 0 to 1, so that contexts of different lengths can be compared on the same graph; at the same time, report different lengths L and distractor strengths d separately, because the position effect is likely to change with these two conditions, and mixing them together for averaging would blur the curves for different conditions. In addition, randomize the order of irrelevant documents, control the total token count of the context, and record the model version and decoding parameters: if the order of irrelevant documents is fixed, position differences may be confounded with differences in document content; changes in total token count alter density; different model versions and decoding parameters may make the whole system behave differently.

Finally, the decision boundary: comparing only one beginning sample and one middle sample cannot rule out sample difficulty and chance fluctuation. A single sample's difference may be caused by differences in difficulty between these two specific questions, unrelated to position. To obtain a credible position curve, one must rely on repeated measurements of paired questions at multiple positions, rather than one or two isolated comparisons.

Lost in the Middle has been repeatedly observed, but “why it happens” does not have a single answer that holds for all models. Research confirms the phenomenon of position bias itself, while its causes may simultaneously arise from three levels—training distribution, model mechanisms, and task design—and the proportions of these three may be completely different across systems.

When diagnosing causes, the inputs should include three types of material: an already reproduced position curve, attention or retrieval traces, and controlled ablation experiments; the output consists of candidate explanations that still hold up after being tested against this evidence. The basic diagnostic method is to change suspicious factors one by one and observe how the position curve changes: change the distribution of context lengths during training to see whether the training distribution shapes the model's preference for positions; change the similarity of distractors to see whether the drop in the middle segment is mainly caused by competition among similar documents; change the structure of the prompt to see whether the prompt's failure to clearly indicate where the evidence is located amplifies the effect. If the curve changes after a factor is changed, that factor is more likely to be part of the cause; factors that leave the curve unchanged after being changed can be downgraded from the candidate list. This process gradually narrows the range of causes rather than finding the single culprit at once.

The possible factors proposed so far fall roughly into four categories. First, primacy/recency bias: during training, models more often encounter distributions in which key information appears at the beginning or end, thereby forming a preference for first and last positions. Second, difficulty of long-distance retrieval: information in middle positions is farther from the current generation step, so the model must cross a longer distance to retrieve it. Third, competition from similar distractors: the more material in the context that is similar to the evidence, the more easily the middle evidence is “drowned out” by other documents. Fourth, prompt structure problems: if the prompt itself does not clearly indicate the structure and location of the evidence, the model can only search the entire context on its own, leaving more room for error. In addition, different architectures and training methods themselves will change performance; even if two systems have the same window size, the shape and causes of their position curves may differ.

The role played by attention weights in such diagnoses needs to be carefully delimited. Attention weights can tell us where in the context the model placed its attention when generating a certain token, and therefore can serve as diagnostic clues; but they are only a correlational observation and cannot alone prove a causal narrative such as “the model forgot the middle because a certain attention head failed.” A lower weight of an attention head on middle positions may be the cause of forgetting, the result of forgetting, or merely an unrelated phenomenon that happens to co-occur. Only by combining ablation experiments that change model structure or input conditions and then observing behavioral changes can we move closer to causal conclusions.

Therefore, we should guard against over-explanation: reproducing the phenomenon does not mean there is a single mechanistic answer that applies to all models. The same Lost-in-the-middle curve may be caused mainly by training distribution in one model, mainly by distractor competition in another, and by a combination of prompt structure and retrieval mechanism in a third. When facing a specific system, the right question is “which factors are at work in it and how much each contributes,” rather than applying a one-size-fits-all mechanistic explanation.

There are only two core strategies for reducing the risk of key evidence being drowned out: minimize irrelevant context as much as possible, while making the task and key evidence easier to locate relative to each other. The larger the window, the more can be crammed in, but the more crammed in, the more the information in the middle is prone to being diluted and obscured. Therefore, the engineering effort should not be aimed at "putting more things in," but at "making the parts that need to be used among what is put in easier to find."

A typical mitigation pipeline takes as input the raw candidate documents, the current question, and a token budget, and outputs a compact context that has been deduplicated, reranked, and annotated with source markers. The pipeline first filters to remove content unrelated to the current question, reducing the number of distractors; then reranks to place high-value evidence near the task, avoiding the weakness of the middle position; and at the same time marks the retained content with source labels, so that every conclusion generated by the model can be traced back to a specific passage. The effect after this processing should be interpreted as "evidence is easier to locate," not "the model's window capability has been permanently changed"—the window and the model themselves have not been upgraded by this input organization; the change occurs only in how this particular input is constructed.

This pipeline has one irrecoverable stage: retrieval. If the retrieval stage misses the correct evidence, then subsequent reranking and position adjustments cannot remedy it—no matter how you arrange things, the correct answer is simply not in the context, and the model cannot fabricate it out of thin air. Therefore, the effectiveness of mitigation measures has an upper limit: they can only make evidence that has already entered the context easier to find, and cannot replace a retrieval stage that can fish out the correct evidence.

Concretely, first retrieve, rerank, and deduplicate; assemble only the passages truly needed to solve the current question, rather than dumping the entire batch of material as-is; use headings, numbering, source identifiers, and separators to establish a clear structure, avoiding the concatenation of long documents into a "text wall" that prevents the model from distinguishing evidence from noise. In terms of position, you can restate the question after the evidence, bringing the task and the evidence closer together in the context; or place the highest-value evidence directly near the task. As for whether to place it before or after the task, and how close is best, there is no universally applicable answer; the specific position should be determined by experiment—different models and different tasks have different sensitivities to position.

For complex tasks, they should be decomposed into multiple stages such as retrieval, verification, and synthesis, with each stage retaining intermediate citations, rather than expecting the model to read the entire context once and directly produce a final answer; decomposition makes each step face a small and clear goal, and the chance of intermediate evidence being drowned out decreases accordingly. For very long materials, first generate a traceable index, letting the model use the index to locate the relevant passages before reading them, rather than directly generating an unverifiable summary—a summary compresses information and also severs the connection between the conclusion and the original text.

Whether a mitigation approach is usable depends on whether it still holds under sufficiently rigorous validation conditions. The easiest mistake is to test only the ideal layout where "the answer happens to be placed at the very beginning"—in that case no approach looks bad, because it avoids the most problematic position. Credible validation must cover combinations of position, length, and distractors, especially scenarios where middle positions, long contexts, and highly similar distractors occur simultaneously.

The input to validation is the complete results of the baseline and the candidate approach on the same batch of position, length, and distractor combinations; the output is the bucketed quality metrics, citation status, latency, and cost differences. Here "same batch" is critical: the baseline does not use the approach, the candidate uses the approach, and apart from that the questions, data, and evaluation method must be exactly the same; only the mitigation approach itself may vary. At the same time, save the final complete request, and afterwards check whether the evidence set was assembled into the context as expected—if the actual request sent does not match the design, the evaluation results cannot be trusted.

There is one hard decision rule: even if the average score rises, as long as the performance of any high-risk position bucket drops, the approach still cannot pass. The improvement in the average score may come entirely from the head positions that were not problematic to begin with, while the middle positions—exactly the places the approach is meant to address—instead get worse. For a mitigation approach, the worst bucket's performance is more informative than the average.

The validation process can proceed in order: first establish a position baseline by measuring the native performance of evidence at different positions without applying any approach; then introduce a length gradient and gradually lengthen the context to see how the effect evolves with length; then add similar distractors to simulate real scenarios where evidence is mixed with noise; then apply a reordering or structuring approach; compare accuracy and citation hit rate before and after application; and finally run regression separately by model version to confirm that the conclusion still holds on new versions, not only on old versions.

When recording metrics, all four of accuracy, whether the evidence citation hits, latency, and token cost must be kept. If accuracy rises but the citation is wrong, it means the model may not have actually found the evidence but guessed the answer correctly based on prior knowledge—such "correctness" is not trustworthy in validation. If the approach works only on a single prompt template and fails with a different structure, it is likely only overfitting that one prompt format rather than truly mitigating the position effect.

The Lost-in-the-middle Effect is an attribution conclusion with strict meaning, not a catch-all term for all failures in long contexts. If the evidence was never retrieved, was truncated before entering the context, used the wrong version, or the question itself is too vague to locate an answer, then the root cause of the failure is likely not positional at all—calling a data pipeline failure “Lost in the Middle” only uses a fashionable term to mask the real problem.

The input to attribution troubleshooting consists of three types of material: the complete request from the failed run, the content finally assembled into the context, and the retrieval and document parsing logs. The output is a judgment of the failure type: either a data pipeline failure or a problem in the model’s use of position. The judgment criterion is strict: only when it is confirmed that the correct evidence is indeed present in the final context and that changing only its position changes the outcome can the failure be interpreted as the Lost-in-the-middle Effect; if the controlled experiment “everything else unchanged, only the evidence position moved” cannot be constructed, then “cause undetermined” should be retained instead of rushing to a conclusion.

Troubleshooting should proceed in a fixed order, first ruling out pipeline problems and only considering the model at the end. Step one: confirm that the final request actually contains the correct evidence; if the evidence is not there at all, the model is bound to answer incorrectly, and position is irrelevant. Step two: check token truncation; content may be truncated when it is too long, or key passages may fall exactly into the discarded portion. Step three: check permission filtering and document parsing; access control may have filtered out the evidence, and the parser may have lost key passages when converting PDF or web pages. Step four: check retrieval and ranking; did the retrieval stage pull out the correct document, and did ranking place it into the final assembly? Only when all these checks pass—the correct evidence is indeed in the final context, and no truncation, filtering, parsing loss, or retrieval omission occurred—can a position-permutation experiment be used to determine whether this is a model-side utilization problem.

Such an order can reclassify many failures that “look like Lost in the Middle” into where they truly belong. A common lesson is: users complain that “the model always fails to find the content in the middle of a long document,” but after troubleshooting it turns out that the parser dropped the middle pages, and the model never saw that passage from beginning to end. Calling this kind of problem “Lost in the Middle” both misses the opportunity to fix the pipeline and falsely inflates the model’s position problem.

Position effects are not something you can test once and be done with. A change in any part—model upgrades, prompt adjustments, retriever updates—can alter the system's effective context, and may even cause a previously inconspicuous Lost-in-the-middle Effect to reappear. Therefore position testing cannot remain a one-time evaluation; it must enter continuous evaluation: treat position sensitivity as a product quality metric tracked over the long term.

The input to continuous regression is a fixed, permutable set of tasks, plus the new system version for each release; the output is trend curves and alerts bucketed by position, length, task type, and risk level. After each upgrade, rerun everything using exactly the same assembly process and scoring process as the baseline, then compare with the baseline. On judgment, there is one non-negotiable rule: even if the overall average score remains stable or even rises, as long as a key position bucket falls below the minimum threshold, the release must still be blocked. The average score cannot hide local collapses, and position buckets are precisely where local collapses most often occur.

The test set itself also needs to be updated regularly. If you always use the same batch of questions and the same templates, both the approach and the evaluation will unknowingly overfit to this fixed format; the measured "stability" is only stability to the template, not stability to real inputs. Continuous rotation of the test set is intended to keep the regression always facing fresh and diverse input distributions.

The test set should retain several types of position-permutable tasks: multi-document question answering (the evidence for the same question can be moved among multiple documents), key-value retrieval (finding the value corresponding to a specified key in a long list), and multi-hop tasks (requiring synthesis of information from multiple places in the context to answer). These task types cover different difficulties from simple localization to cross-span synthesis. After each upgrade, plot the bucketed position curve, not just an overall average score—only the curve can reveal which segment of positions is degrading.

For high-risk answers, the model should also be required to provide cited evidence as a mandatory output in addition to the answer. When citations are missing, or when the citations and the answer contradict each other, trigger supplemental retrieval or manual review, instead of letting an answer without evidential support flow out directly. This mechanism further transforms the Lost-in-the-middle Effect from an evaluation metric into an online safety net: even if the model fails to find evidence at a certain position, the system can intercept it through citation verification, rather than letting it answer based on prior guesses.

Let's use a concrete scenario to tie together the earlier measurement methods: suppose there is a customer-service task where only one of 30 equal-length documents contains a refund exception clause, and the answer is hidden in that document. Place this evidence at the beginning, middle, and end of the context respectively, keep the other 29 documents unchanged, and then record the answer accuracy at each position. The question is: is the resulting curve difference a true position effect or random sample fluctuation?

Figure 1 shows a schematic U-shaped position curve: the horizontal axis is the relative position of the evidence in the context, and the vertical axis is accuracy; the red line represents the results under the original long context, with a clear dip at the middle position; the green dashed line represents the results after retrieval reranking, with the middle dip lifted. It must be emphasized that this figure is schematic: the shape of the real curve must be measured on a specific model, specific length, and specific task, and this figure cannot be treated as a universal constant applicable to any system.

Figure 1: Schematic U-shaped position curve. The red line is the original long context, and the green dashed line is the result after retrieval reranking; the real shape must be measured on a specific model, length, and task, and this figure cannot be treated as a universal constant.

The accompanying numerical example can be organized into the table below. Each row is one position condition, and the number correct is based on a total of 60 samples:

The interpretation of the numbers should be done step by step. The original middle section is 34/60 ≈ 56.7%, and after reranking it is 46/60 ≈ 76.7%, an absolute improvement of about 20 percentage points. On the surface the reranking effect looks significant, but there is a trap in attribution: if 8 distractor documents are also deleted at the same time as the reranking, then part of that 20-percentage-point improvement comes from the reduction in interference, and only another part comes from the position shift; the two cannot be separated. At this point you cannot attribute the entire improvement to 'the evidence being placed further back'; you should run an ablation experiment that changes only the position and leaves the document set untouched, to isolate the contribution of the position variable alone.

A more robust approach is a paired experiment: for the same question instance, generate three versions with the evidence at the beginning, middle, and end, keeping the evidence content, distractors, total length, and answer exactly the same, with the only difference being the position of the evidence. In this way each question serves as its own control, and differences in question difficulty are canceled out by pairing, allowing the position effect to emerge from the question noise. If the task requires combining two pieces of evidence scattered in the context to answer, you should also separately test layouts such as 'both in the middle', 'one at the beginning and one at the end', and 'both adjacent to each other'—failure in multi-hop synthesis is not necessarily equivalent to failure in single-fact retrieval, and the curves for these two types of tasks may have completely different shapes; mixing them together in testing will contaminate each other.

With a small sample size, fluctuations in percentages are likely to be purely accidental. For example, in 60 samples, getting 2 more or fewer correct causes the accuracy to fluctuate by about 3.3 percentage points; with even smaller samples the fluctuation is larger. Therefore curves based on small samples must be accompanied by confidence intervals and replicated across multiple different question wordings to see whether the effect holds under different formulations.

Product thresholds should be set according to risk level. Ordinary recommendation scenarios can tolerate slight position sensitivity; but high-risk policy interpretation and compliance answers should meet minimum accuracy and citation hit rates in every position bucket, and any bucket that fails should be considered unqualified. A higher overall average must never mask middle-section failures: if sampling includes many beginning samples and few middle samples, the total score will be biased toward the well-performing beginning, and the most dangerous positions will disappear from the report. Therefore at least bucket by the three dimensions of relative position, context length, and interference intensity, and report sample sizes or confidence intervals for each bucket, rather than reporting only a single overall average.