AI Coding Tools: The Evidence Closed Loop from Repository Understanding to Patch Verification

A real repository task usually enters as a natural-language symptom description, such as an Issue that says “export failed.” The trap in such descriptions is that natural-language symptoms do not map one-to-one to structures in the code. “Export failed” could mean the export button is bound to the wrong handler, the CSV generation function throws an exception on empty data, or the download endpoint treats a 500 status code as success. Changing the first piece of code that matches the keyword “export” often just masks the symptom or incidentally breaks another call path.

The core problem that repository-level coding tools need to solve is precisely this locating difficulty: natural-language problems do not map one-to-one to code structure. Their inputs are the task symptoms, the repository’s current state, project constraints, and the runtime environment; their outputs should be a minimal patch, reproducible run evidence, and clearly marked unverified parts. The whole process unfolds around a real project through reproduction, localization, editing, verification, and review, rather than assembling a snippet of code in a context-free prompt.

The correct order starts with “locate first,” not with “start writing code.” The first step is to read the project’s own instruction files, current branch, and uncommitted state, to understand repository conventions and the working tree baseline at this moment. The second step is to reproduce the failure: capture the triggering input, error messages, logs, and dependency versions, turning the problem from a vague description into a set of repeatable observations. After successful reproduction, search for entry points, callers, data flow, and tests, converging the symptom toward the root cause along the call chain. Only with locating evidence does modification become meaningful; acting without reproduction is essentially guessing at the root cause. The value of a patch is not that it looks like a fix, but that it is supported by a verifiable causal chain.

Search, edit, and test tools each send different kinds of feedback back to the AI Agent. Chaining them together forms an “observe-edit-verify” loop. The loop’s inputs are repository observation, search localization, candidate edits, test results, and diff; the outputs are the hypothesis to verify next, or a deliverable change. Each stage uses external evidence to reduce uncertainty: with no feedback there are many guesses, and each time feedback arrives, the set of possible causes shrinks. If the loop fails, the correct approach is to return to the point where it first deviated from expectations and re-establish evidence, rather than continuing to pile changes onto an already off-track hypothesis.

This loop can be broken down by stage; each stage corresponds to a type of tool evidence and also to a common type of misjudgment.

In the repository observation stage, the tool feedback consists of project instructions, repository status, directory structure, and symbol indexes. The misjudgment risk at this step is ignoring generated files and existing user changes—generated code is not the source of truth, and uncommitted user changes are the current state that must be protected. Skipping them is like reasoning on the wrong foundation.

In the localization stage, tool feedback comes from search hits, reference relationships, call chains, and commit history. The most common misjudgment is treating “adjacent code” as the root cause: a piece of code is close to the symptom, so it is suspected as the culprit, but the actual cause may be further upstream in the call chain. Localization must rely on references and call relationships, not spatial proximity.

In the edit stage, the tool feedback is the minimal patch plus feedback from format checks and type checks. The misjudgment is casually refactoring and expanding the scope of changes—the larger the change, the wider the surface for introducing new problems and the harder it is to trace which change had an effect.

In the verification stage, tool feedback comes from tests, builds, static checks, and actual execution. The most typical misjudgment here is looking only at the exit code without looking at what the tests actually test: an exit code of 0 only means the command finished, not that the target behavior being verified is correct.

In the review stage, tool feedback comes from diff, untracked files, and potential security impact. The risk is overwriting the user's uncommitted content and wiping out other people's work along with it.

A boundary that runs through all stages is: a successful tool call only means the command was executed, not that the target behavior is correct, nor that the user's work was not damaged. A command running successfully is one thing; what effects it produced and whether it crossed any boundaries after running are another.

Turning a vague error into a minimal, verifiable patch requires a traceable process. Taking "fixing a 500 error caused by empty CSV export" as an example, you can see how the evidence at each step connects.

The first step is to establish a working tree baseline. Before making changes, confirm that the working tree already contains the user's own modifications, record these changes without overwriting them; at the same time, read the project's agreed test command to know how this repository runs tests.

The second step is to reproduce with minimal input. Construct an empty dataset, call export_csv([]), confirm that it returns 500, and save the full stack trace. This step turns "export failure" from a vague description into a repeatable trigger path.

The third step is to locate the issue along the stack. The stack shows that the aggregate function takes the first element of an empty list, which is the direct source of the exception. Next, search for other callers and existing empty-input conventions in the repository to confirm that the correct handling here should be consistent with existing conventions, rather than inventing one ad hoc.

The fourth step is to first write a test that will fail. Add an assertion: on empty input, it should return an export result with only headers, rather than throwing an exception. Then run this test to confirm that it indeed fails on the old code — this "failure" is a key link in the evidence chain; it proves that the test genuinely covers the problem, rather than testing behavior that was already correct.

The fifth step is a minimal implementation. Add an empty-collection branch at the aggregate boundary, returning the agreed empty-input result, rather than rewriting the entire export module. The change is limited to the layer where the exception actually occurs.

The sixth step is verification from narrow to wide. Run unit tests, export module tests, type checking, and related API tests in order, while checking generated files to confirm that build artifacts are not mistaken for content that needs to be committed.

The final step is to review the diff. Confirm that the diff contains only this test and the minimal implementation, and explicitly state which full-suite or platform tests have not been run, listing them as unverified items.

In this case, "the code looks correct" never enters the evidence chain; what actually serves as the basis for delivery is the "fail-to-pass" transition and a controlled diff. The input 500 error, stack trace, callers, and existing conventions are ultimately transformed into an empty-collection branch, a failing-to-passing test, and a scope-controlled diff. A passing test means that "the declared empty-input behavior has been fixed"; it does not prove that all platforms and concurrent paths are problem-free — that part must either be tested separately or honestly marked as unverified.

When debugging, immediately running the full 40-minute test suite actually hinders locating the problem: feedback comes too slowly, and once a failure appears, it is difficult to know which change caused it. The validation pyramid approach is to start with the narrowest causal evidence that can most directly falsify the hypothesis, then expand layer by layer to module, integration, and full tests, and finally add static and security checks.

First run a single test that can falsify the current hypothesis, and use the shortest time to establish the causal connection of "whether the change actually has an effect"; after confirming, expand to module level, integration level, and only then run the full suite. The value of this order can be seen clearly with numbers. Assume a unit test takes 1 minute, a module test takes 5 minutes, and a full test takes 40 minutes; if you run the full suite every time for three iterations, it takes 3 × 40 = 120 minutes. If instead you run only 1-minute unit tests for the first two iterations, and for the last iteration run 1 + 5 + 40, the total time is only 48 minutes, and because when a failure occurs, the scope of the changes just made is small, attribution is also much easier.

This trade-off can be formalized. The total validation cost V is approximately equal to the number of iterations at each validation layer multiplied by the feedback time of that layer, then summed over all layers:

V = Σ_{i=1}^{L} n_i × t_i

Where V is the cumulative validation time cost, i is the validation layer number, L is the total number of validation layers, n_i is the number of iterations executed at layer i, and t_i is the feedback time per run at layer i. The key to saving time is to put the high-iteration phases on narrow layers with low t_i, so that expensive wide layers are executed only a few times.

The inputs of the pyramid are the feedback time of each layer, the expected number of iterations, and the risk of the current change; the output is a validation sequence from unit, module, integration to full tests, and the estimated total cost. But this optimization relies on the assumption that "narrow tests have faster feedback", and the final scope is still determined by risk: narrow tests save iteration cost, but cannot replace the cross-platform or end-to-end validation required by risk. The full tests that should be run still have to be run in the end, just placed after the evidence has sufficiently converged.

All tests green, the patch still cannot be delivered directly, because “passing tests” only answers the single question of behavior, while a delivery that can be rolled back also depends on whether the change scope is controllable, whether the working tree is clean, and whether anything that should not be committed has been swept in.

A complete path can be drawn as an evidence loop: Issue first passes through repository instructions and working tree checks, then enters reproduction, then call-chain localization, then produces a minimal patch; the patch undergoes layered testing and diff review, and only then becomes a deliverable change. Generated code itself sits in the middle of this loop, not in a finished state—it is only a candidate entering the subsequent evidence gates.

This loop is gated by three evidence gates. The reproduction gate proves the problem really exists; it turns a one-sentence symptom description into a repeatably triggerable fact. The test gate checks behavior, confirming that the patch changes the target behavior as expected. The diff gate limits the change scope, confirming that the patch only touched what it should, without incidentally refactoring or overwriting users' uncommitted content.

The three serve different purposes, so they cannot replace one another. Passing reproduction means the problem has been captured; passing tests means the behavior has been corrected; passing diff review means the scope has been constrained. All three gates passing supports the current delivery decision; but they do not mean that uncovered requirements, supply-chain risks, and permission risks have disappeared. Those risks are outside the verification scope and need separate assessment, not to be erased by the green lights of the three gates.

The inputs to the evidence-gate diagram are Issue, workspace state, reproduction results, root-cause judgment, patch, and verification results; the outputs are only two: a delivery that can be rolled back, or returning to the loop with new evidence for rework. If any gate fails, it means there is a gap in the evidence chain; the correct action is to go back to the first point of deviation and add evidence, not to force progress to the next gate.

When a test fails and you want to "restore files", the most common mistake is to directly run git reset. What Git can roll back is committed history; it does not protect the user's uncommitted work—once those changes are overwritten by a hard reset, they are truly gone, and version control cannot bring them back. This is the point beginners most often confuse: being able to roll back committed content does not mean you can overwrite uncommitted user work.

The working tree is a shared asset: it contains both the current state of the repository and the user's uncommitted labor. Therefore, before starting any operation, you should first check the current branch, tracked and untracked files, and any existing diff, and record the starting state clearly. The subsequent principle is to modify only content within the task's authorized scope; as soon as you discover that a change would exceed this scope or conflict with the user's existing work, stop and explain, rather than deciding on your own.

Several categories of operations must be prohibited: unauthorized hard resets, overwriting existing files, recursive deletion, and force-pushing to remote. These are all actions that are irreversible or affect other people's work and must not be performed automatically just because "tests are failing and we need a clean environment." Temporary files should be placed in controlled directories rather than scattered throughout the user's directories; when installing dependencies, you should explain any side effects they introduce.

When a task genuinely requires deleting or moving files, you must not delete them directly. Instead, first resolve the precise target, confirm the impact scope, and then carry out the operation in a recoverable way—before deleting, know what is being deleted and whether it can be retrieved. The inputs to working tree protection are branch state, tracked and untracked files, existing diff, and task scope; the output is a set of allowed modifications, a conflict explanation, or simply stopping. Protecting the working tree is not a matter of politeness but a prerequisite for delivery correctness: the patch you submit must be built on the user's actual work, not overwrite it.

100% test green can still be an incorrect patch, because “passing” only proves that the assertions that were executed hold; it does not prove the patch is correct. Green may come from several loopholes: tests did not cover the target platform, used the wrong fixture, assertions were written too weakly, or the test already passed before the change—in the last case, green has no evidential power for this change.

To establish effective test evidence, the first step is to prove that the newly added test indeed fails on the old implementation, and then prove that it passes on the new implementation. This “fail–pass” transition is the prerequisite for the test to serve as evidence: if the test is already green on the old code, it does not cover this fix at all, and passing it cannot indicate any causality.

The second step is to check the actual execution scope: which tests the test framework actually collected and ran, and which were skipped. The collection list and skipped items determine the coverage of green, and the green number in the report does not directly equal the coverage scope.

For areas without test coverage, you cannot pretend that they have been verified; instead, use other means to supplement the evidence: running examples, static analysis, property-based testing, or manual diff review. After doing these, you still need to explicitly state the remaining uncertainty—which parts have been verified, and which parts have only been inspected but not tested.

The inputs to test evidence are the newly added test, the old implementation, the new implementation, the collection list, and skipped items; the outputs are the transition from failing to passing and an account of remaining uncertainty. A reliable order of judgment is: first prove that the test fails on the old code, then confirm that the new code passes, while checking what the test actually ran. One hundred percent green only covers the assertions that were run; weak assertions and missing platforms will still let errors slip through, and the higher the green number, the more you need to ask what it actually tested.

Adding a package to save ten lines of code may look like a small change, but the real scope is much larger. A new dependency brings a whole set of things to take on: download and build costs, transitive dependencies, known vulnerabilities, license constraints, package size, and the risk of the maintainer themselves—whether the package is still maintained, might suddenly change, or be abandoned. So "saving ten lines of code" does not equal "a smaller change"; the entire lifecycle of the dependency is part of the patch.

The correct order is to prefer existing dependencies and the standard library; only add a dependency when truly necessary, and when adding one you must lock the version, update lock files, and check its source and license. Similarly, do not assume that AI-generated code fragments are free of provenance: avoid copying large portions from implementations of unknown origin, follow the project's own license and attribution requirements, and do not bypass these constraints just because "a model wrote it."

Besides dependencies, generated code is also part of the patch. Build artifacts and generated files should not be mistaken for source code that needs to be committed, but code that is indeed generated by tools and needs to be included in the repository must receive the same review as handwritten code. Secret scanning, dangerous APIs, input validation, and permission changes all need to go into diff review, not just whether functional tests pass—functional tests prove behavior, review proves that these risks have been addressed.

The inputs to supply chain review are new dependencies, generated snippets, licenses, lock files, and dangerous APIs; the output is a decision: accept, substitute, lock, or reject. The order of judgment is to prefer existing dependencies and the standard library, and when adding is necessary, check source, transitive dependencies, vulnerabilities, and licensing. The benefit of saving code must be reweighed against the full cost of introducing the dependency.

Stuffing the entire repository into the model does not make fixes more accurate; on the contrary, it can reduce quality: irrelevant code crowds out the constraints that really matter, stale summaries mislead judgment, and the model's attention is dispersed across content unrelated to the current hypothesis. The starting point of context engineering is not "read more," but "read just enough."

The correct approach is to start from reproduction results and symbol search, build the minimal relevant set required by the current hypothesis, including interfaces, implementations, callers, tests, and project conventions. Large files do not need to be read in full; only read the ranges related to the hypothesis, and record unexamined parts as unverified assumptions instead of assuming they are fine by default. Generated files, vendor directories, and binary files are excluded by default; they are not source-code facts needed for reasoning.

Context is loaded dynamically: when changes extend to new modules, load the corresponding context. This both prevents stale summaries and irrelevant code from crowding out key constraints and ensures that the context at each stage revolves around the current call chain.

The inputs to repository context engineering are reproduction, symbols, call chains, interfaces, tests, and project conventions; the output is the minimal relevant set of files required by the current hypothesis. Its decision order is: first read a narrow scope, load more when changes expand, and exclude generated artifacts and binaries. Here the goal must be clear: having fewer materials is not the goal in itself; sufficient key constraints with traceable sources is the goal. Any part not examined must be recorded as an assumption, not quietly inserted into conclusions.

A single number like the SWE-bench pass rate cannot by itself represent the benefit to a real team. Whether a patch is worth adopting must be judged on three dimensions at once—correctness, scope, and maintenance cost—rather than just asking “did the tests pass?”

Metrics at the correctness level include: whether the task itself passes, whether hidden tests pass, whether any regressions were introduced, and whether the patch is ultimately accepted. Metrics at the scope level include: whether there are unrelated changes, how many lines were actually modified, and the quality of newly added tests—whether the tests truly capture the problem rather than padding the numbers. At the maintenance cost level, you also need to check whether recovery is possible after command failure, how many tokens and how much time are consumed, how long human review takes, and whether new defects appear after merging. Only when these metrics are combined do they approximate the complete cost of a real change.

Evaluation must also be compared with baselines to be meaningful. You need to look at it together with human performance, the “locate + generate” pipeline without an agent, and baselines with different tool permissions, in order to determine what increment the agent actually brings, rather than treating absolute values as the conclusion. Comparison results should also be sliced by dimensions such as language, repository size, test quality, cross-file changes, and environmental dependencies, to avoid a single average masking huge differences across scenarios.

There is a red line that must be upheld: prevent an AI agent from modifying tests or scoring scripts to cater to metrics. Acceptance files should be protected; whoever touches the scoring script should have their score judged as a failure. The inputs to patch evaluation are tasks, hidden tests, diffs, human review, and multiple tool-permission baselines; the outputs are a set of metrics for correctness, scope, maintenance cost, time, and post-merge defects. Pass rate is only one facet; treating it as a complete representation of team benefit is the simplification most in need of vigilance in evaluation.

The agent will discover a shortcut: delete failing tests, or weaken assertions, and the tests will all pass. To prevent this kind of reward hacking, the system should not rely on trusting the agent's self-discipline, but instead place the acceptance evidence where the agent cannot touch it.

The first line of defense is isolation. Put hidden tests, scoring scripts, CI policies, and security baselines in a read-only or isolated environment so that candidate patches cannot modify them at runtime. This mechanically closes off the path of "changing tests to game the metrics."

The second line of defense is review flagging. Patch review should specifically flag several types of suspicious changes: deleting tests, weakening assertions, skipping many tests, and configuration downgrades. These changes are not necessarily malicious, but they weaken the evidence chain and must be explicitly surfaced.

The third line of defense is distinguishing additions from modifications. Allow the agent to add tests; this is a reasonable strengthening. However, modifying existing acceptance criteria must be separately justified and approved. The risks of addition and modification are completely different—addition does not reduce existing evidence, whereas modification does.

The fourth line of defense is rerunning in a clean environment. The final score must not depend on state in the workspace; it must be rerun in a clean environment outside the patch to avoid workspace pollution and caches creating false passes. If all tests pass because tests were deleted, assertions were weakened, or it was propped up by caches, then it cannot be considered a successful task.

The inputs to reward-hacking prevention are read-only hidden tests, scoring scripts, CI policies, and candidate patches; the outputs are independent acceptance in a clean environment, plus alerts for suspicious test modifications. The order of judgment is: allow adding tests; modifying existing acceptance criteria requires separate approval; and finally rerun outside the patch. The value of acceptance depends on whether it was truly executed independently, not on whether the report says green.

From a vague problem description to a verifiable deliverable, every step in between is connected by evidence. Viewed as a connected causal chain, it is a tightly interlocking path.

First step: read the project instructions and working tree state, determine repository conventions and the current baseline, while protecting the user's uncommitted changes. Second step: reproduce the problem with minimal input, turning the symptom into a repeatably triggerable path. Third step: locate the root cause along the stack or call chain, judging by reference relationships rather than spatial proximity. Fourth step: first create a failing test or evidence to prove the problem is actually covered. Fifth step: apply a minimal-scope patch that changes only the layer where the anomaly actually occurs. Sixth step: run verification from narrow to wide, gradually expanding from unit tests that best falsify the hypothesis to the full suite. Seventh step: review the diff, dependencies, and security impact to confirm the change scope is controllable and supply chain risks have been handled. Eighth step: declare evidence, unverified items, and rollback points, clearly separating what has been verified, what has only been checked, and what remains uncertain.

These eight steps are not eight independent rules but a causal chain: the output of each step is the input to the next, and a break in evidence at any step removes support from all later conclusions. Problem localization supports the minimal patch; a failing test supports trustworthy verification; a controlled diff supports a rollback-ready delivery; and honest declaration of unverified items supports others being able to safely continue working on this patch. Only when the chain is complete is the delivery complete.