Model Evaluation and Benchmarks: Turning "Stronger" into Reproducible Evidence

# Clarify Exactly What Is Being Estimated

Does a single model have a "true ability score" that is independent of context? No. Any number produced by an evaluation is not an intrinsic property of the model, but rather an estimate of performance under the combined influence of a particular question distribution, a particular prompt template, particular tool permissions, a particular inference budget, and a particular scorer. Treating this estimate as a fixed ability of the model is the easiest and most expensive mistake in evaluation.

Take a refund assistant as an example. When we place it in a scenario, what we really care about are several mutually independent behaviors: whether policy Q&A is accurate, whether cited evidence is adequate, whether amount calculations are error-free, whether it refuses when it should refuse unauthorized requests, and whether it can clarify over multiple turns when information is insufficient. These five abilities each have their own failure modes. A generic knowledge leaderboard cannot cover all of them simultaneously, nor can it replace direct measurement within the scenario. Conversely, a refund assistant's rank on a generic leaderboard cannot be directly converted into its performance in real refund conversations—if the questions change, the tools change, or the evaluation criteria change, the score may shift as a whole.

Therefore, rigorous reports never write "Model M has a score of 82," but rather "an estimate of metric Y for model M on test set D under configuration C." Every item in this sentence carries actual weight. The model name is only part of the experimental subject; the system prompt, temperature, maximum token count, reasoning effort, number of candidates, and available tools all belong to the treatment conditions. The same weight snapshot at temperature 0 and at temperature 1 yields two different experimental results, not the same ability measured twice. Raising the temperature introduces randomness; increasing the number of candidates means the model can make several guesses and then choose; enlarging the inference budget allows a longer chain of thought—all of these change the final score, but they change the measurement conditions, not the object being measured itself.

The cognitive anchor should be fixed: evaluation is not taking a photograph of an ability, but measuring future usage performance with a limited sample. The photograph-like view makes people believe the score is intrinsic to the object; the measurement-based view reminds us that the sample may not be representative, the instrument itself may be biased, and what we get is always an estimate with uncertainty.

Write this principle as an input-output relationship: the inputs to model evaluation are model snapshot M, test distribution D, inference configuration C, and target metric Y; the output is an ability estimate for Y with uncertainty. A score is valid only for the given questions, prompts, tools, and budget; it is not a context-independent true ability constant. After the question distribution or inference configuration changes, old scores cannot be directly extrapolated—to answer a new question, you must re-estimate under new D and C.

# Model evaluation and application evaluation cannot substitute for each other

The bare model answers better, so why might it still perform worse after being built into a product? Because the product places the model in a different pipeline: retrieval first proposes candidate clauses, reranking decides which ones enter the context, system prompts constrain permissions, business processes decide when it must refuse and when to transfer to a human, and users also care about latency and citation trustworthiness. Answering questions correctly is only one link in this chain; a high score on that link does not mean the whole chain is fault-free. To understand this gap, we need to break evaluation into four levels, each with different things fixed and different questions answered.

The first level is model evaluation. It fixes the prompt, tools, and question set, and only has candidate models compared with each other under these same conditions, answering "which one is better at this type of question." In the refund assistant, the typical question at this level is: given the same policy passage, which of Model A and Model B judges "whether a refund is possible" more accurately. Because all conditions are fixed, score differences can only be attributed to the model itself.

The second level is component evaluation. It reverses this by fixing the model, and each time replaces only one component in the pipeline, answering "does this component bring improvement?" For example, keeping the model unchanged, swap in a different retrieval or reranking scheme to see whether the correct clause can enter top-5. If the top-5 hit rate rises, the improvement comes from the retrieval side; if it stays the same, the bottleneck is elsewhere. The reason this level can locate the source of a problem is that only one variable is changed at a time.

The third level is application evaluation. It runs the complete version within the business process, answering "does the end-to-end system achieve its goal?" What is accepted here is no longer answer accuracy, but whether citations are real and verifiable, whether permissions are strictly followed, whether latency is within an acceptable range for users, and whether the final user outcome is qualified. Even if the refund assistant answers correctly, if it cites hallucinated clauses or says compensation it has no authority to promise, application evaluation should judge it as unqualified.

The fourth level is online experiment. It uses traffic-splitting rules to divide real traffic between the new and old versions, with guardrails in place, answering "can offline gains really transfer?" In offline evaluation, the transfer-to-human rate may decrease, but online you must verify whether this decrease actually happens, whether erroneous promises decrease, and whether guardrails intercept abnormal behavior. The gap between offline and online can only be tested at this level.

The division of labor among these four levels forms a clear causal chain: model evaluation isolates capability, facilitating attribution; application evaluation preserves the real pipeline, facilitating decision-making. The two should be used in combination like unit tests and end-to-end tests, not as either/or. Doing only model evaluation misses failures outside the model such as permissions, latency, and retrieval failures; doing only application evaluation cannot locate whether a problem is due to the model or a component.

Write the level analysis as input and output: the input is the object to be compared and the fixed conditions; the output is an evaluation conclusion at one of the four levels—model, component, application, or online experiment. Model evaluation fixes system conditions to isolate candidate capabilities; application evaluation preserves the complete pipeline to judge business results—the former is responsible for attribution, the latter supports release. A high score from a bare model can never replace end-to-end acceptance of permissions, latency, and user outcomes.

# Which Variables Need to Be Fixed for a Fair Comparison

Two reports both say “accuracy 80%”, so why might they be completely incomparable? Because “accuracy” is just a number calculated at the end, while the variables that determine that number number more than a dozen; if any one differs, the meaning of 80% is different. The first step in a fair comparison is to align all of these variables: the question set and its version, few-shot examples, system prompt, output parsing method, context fed to the model, available tools, token limit, stopping condition, temperature, number of candidates, failure retry rules, and the scorer used for final grading. Every link in this chain directly changes the score — a different question version means the model is not seeing the same set of questions; different few-shot examples mean different demonstrations; different output parsing means the same generated text may be read as correct or incorrect.

Budget variables especially require caution. If the purpose of the comparison is cost–benefit, then the inference budget itself is an independent variable; you cannot use cheap single-shot generation on one side and expensive multiple sampling on the other, and then report the most favorable point for each. The correct approach is to draw a quality–cost curve: the horizontal axis is cost or compute, the vertical axis is quality, and the two models are compared across the whole curve. Picking only the most favorable point for each yields not a comparison but two advertisements.

The choice of metric must likewise be decided before the results. A fair experiment pre-registers the primary metric, key slices, and failure threshold, for example declaring in advance: “Using refund-amount judgment accuracy as the primary metric, slicing by amount range and language, and any slice below 80% constitutes failure.” The point of doing so is to prevent picking a more presentable metric after seeing the results — choosing a metric after the fact makes the experimenter both player and referee.

Reproducibility requires fixing the environment as well. The experimental configuration should be saved as an executable script, while recording the model snapshot, data hash, and random seed; if a hosted model is used, at least record the date and version number, because server-side updates may quietly change behavior. The run order should also be interleaved or randomized, alternating the two models, to avoid service updates, rate limiting, or time drift being concentrated on one model.

Failures and timeouts are the places most easily exploited. Retries must be scored according to pre-written rules: whether a timeout counts as wrong or is retried once before scoring must be decided before the experiment begins; silently discarding timed-out samples inflates quality, because the discarded ones are often the questions the model could not answer. The existence of pre-defined rules ensures that the attribution of failed samples is not decided by post-hoc emotions.

The most common sleight of hand hides among these variables: Model A generates one output per question, while Model B generates 100 candidates per question and then picks the best. Even if the final scores are the same, it absolutely does not mean the two are equally reliable under interactive single-shot use — B’s score purchased a large amount of resampling; A’s did not. Calling single-shot generation versus best-of-100 picking a fair comparison is like putting “guessing once” and “guessing a hundred times and selecting the correct one” in the same table.

Write the fair experiment as input and output: the inputs are the same question version, few-shot examples, prompt, tools, token, temperature, candidates, retries, and scorer; the output is comparable per-question results and a quality-cost curve. Different budgets mean different treatment conditions; you cannot compare single-shot generation against best-of-100 picking. Failures and timeouts must be scored according to pre-registered rules, and hosted models must also record the date and version. Without any one of these alignment items, the “comparison” between two 80% scores is merely coincidence.

# Worked Example: How Certain Is the Gap on 100 Refund Questions?

If Model A answers 82 questions correctly and Model B answers 78 correctly, can we directly declare A stronger? First unfold the entire process: start from a fixed task, obtain two point estimates, then perform slice analysis separately and provide confidence intervals, and only then judge whether the gap is stable. Figure 1 shows exactly this chain, and its first message is: a difference in point estimates does not equal a stable difference, and paired results on the same set of questions carry more information than two independent accuracies.

First compute the uncertainty. The standard error of a binomial proportion is SE = $\sqrt{\text{phat × (1 − phat) / n}}$, where phat is the proportion correct and n is the number of questions. A answered 82 questions correctly, phat = 0.82, n = 100, SE = $\sqrt{\text{0.82 × 0.18 / 100}}$ ≈ 0.038. The rough 95% confidence interval is phat ± 1.96 × SE, i.e., 0.82 ± 0.075, about 74.5%–89.5%; calculated with the more precise Wilson interval, A is about 73%–88%. B's phat = 0.78, Wilson interval about 69%–85%. The two intervals overlap substantially. Overlap does not mean 'absolutely no difference', but it shows that with only 100 questions and a 4 percentage point gap it is hard to draw a strong conclusion—this batch of data is compatible both with 'A is truly stronger' and with 'the two are about the same, and the 4 points are just sampling fluctuation'.

If the two models answer the same set of questions, the paired information can reduce the uncertainty. After aligning question by question, there are 12 questions that only A answered correctly and 8 questions that only B answered correctly; on the remaining questions both are correct or both are wrong. The information that truly distinguishes the two models comes mainly from these 20 discordant samples; the concordant correct and concordant wrong questions contribute nothing to the comparison. For this paired structure, one can use the McNemar test to judge whether the 12 vs. 8 difference is significant, or use bootstrap to estimate the interval for the difference 'A-only correct proportion minus B-only correct proportion'. The paired comparison is stronger than independent comparison precisely because it holds the difficulty of each question fixed—on the hard questions both models are wrong, and this no longer gets mixed into the overall gap.

Slice analysis can also overturn the overall conclusion. Look at the high-risk 'unauthorized refund' slice: out of 10 questions, A answered only 2 correctly and B 8. A, with a higher total score, is clearly worse on this slice, and this slice corresponds exactly to a real business risk—once an unauthorized refund is allowed through, its cost far exceeds getting an ordinary question wrong. So the conclusion reverses: A has a higher overall score but cannot be deployed. The lesson at this layer is that confidence intervals under the overall metric and slices can give different answers, and a high-risk slice can independently veto an overall advantage.

Write the accuracy example as input-output: the inputs are sample size n, proportion correct phat, and the same-question paired results; the outputs are standard error SE, 95% interval, and paired difference evidence. SE = $\sqrt{\text{phat × (1 − phat) / n}}$, the rough interval is phat ± 1.96 × SE; A's 82% and B's 78% intervals overlap substantially, showing that one hundred questions are not enough to make a strong assertion. When the question distribution or high-risk slices impose additional requirements, an advantage in overall score must yield to evidence from slices.

# pass@k Mixes Model Quality and Compute Budget Together

A code model reports pass@100 = 90%. Why might a user still frequently fail with a single generation? Because pass@k describes not the “success rate of a single generation” but rather “the probability of finding at least one usable answer among k candidates.” The gap between these two numbers is bought by compute budget.

Let p be the probability of success on a single independent generation. The idealized “at least one success in k attempts” is 1 − (1 − p)^k. Each step of this formula is intuitive: (1 − p) is the probability of a single failure, (1 − p)^k is the probability that all k attempts fail, and subtracting it from 1 gives the probability of at least one success. When p = 0.1, pass@1 = 10%, while pass@10 = 1 − 0.9^10 ≈ 65.1%. The single-shot success rate is only 10%, yet after tenfold sampling the probability that “there is a correct one in the candidate pool” exceeds 65%. The jump in the number comes mainly from the tenfold sampling budget, not from the single-shot model suddenly becoming smarter. This formula is only an ideal upper bound: real generated errors are correlated with each other—the same model tends to make the same kind of mistake on the same problem, so k attempts are not k independent chances; moreover, having “a correct one” in the candidate pool does not mean the system can pick it out—a reliable verifier is still needed to select the correct one from the candidates. The verifier can be mistaken, and the correct candidate may also be discarded.

Precisely because pass@k mixes model quality and sampling budget together, reporting a metric must also answer “what does the user actually get.” pass@1 promises a single-sample hit, corresponding to a real user’s single-click experience; disclosing it requires noting the temperature, token, and prompt. pass@k promises at least one hit in the candidate pool, and must disclose k, total token consumption, and candidate correlation—the larger k is and the more expensive the tokens, the farther this metric is from a single user experience. One level further down is post-selection success rate: the rate at which the verifier actually delivers the correct candidate to the user, which depends on verification cost and the mis-selection rate. The final level is end-to-end success rate: the rate at which the complete flow actually works, counting tool calls, timeouts, and failed retries in the denominator.

The same applies when comparing reasoning models. A fixed-token comparison may be unfair, because the two models have different efficiency at converting tokens into answers; a fixed-price or fixed-latency comparison answers different questions—one asks “who is stronger for the same money,” the other asks “who is stronger under the same speed.” The most honest approach is to report the Pareto frontier: plot quality as a function of budget, let readers choose a point on the curve according to their own budget, rather than choosing one budget for everyone and reporting a single number.

Write pass@k as input and output: the input is the single-shot independent success probability p and the number of samples k, and the output is the ideal probability of at least one success, pass@k = 1 − (1 − p)^k. When p = 0.1 and k = 10, it is about 65.1%; this improvement comes mainly from the tenfold compute budget. Real errors are correlated and a verifier still needs to choose correctly, so the formula is only an upper-bound intuition. Any pass@k report must also disclose k, total token, latency, and the selector, otherwise the reader cannot know how much of this number belongs to the model and how much belongs to the budget.

# Why Public Benchmarks Get Contaminated and Overfit

Can a set of questions be considered “unseen” just because it was never explicitly used for fine-tuning? No. The original problem statements, standard answers, web mirrors, solution explanations, translations into various languages, and even derivative texts generated by other models may all have already entered the pre-training corpus. During pre-training, the model has seen many variants of the problem statement and answer; it is just that those moments of exposure were not recorded in the fine-tuning logs. Thus, between “not used for fine-tuning” and “unseen” lies the entire pre-training corpus.

More insidious is overfitting at the organizational level. Teams repeatedly adjust data mixes and training recipes based on public leaderboards: when the ranking drops a little, they change data, modify prompts, or tune sampling parameters until the score recovers. This process is different from training directly on the test set, but the effect is similar—each time the leaderboard is checked and an adjustment is made, the test set loses a bit of its independent judgment. After many rounds of iteration, a high score may reflect not that the model is better at solving the problems, but that the team is better at fitting to this particular problem set.

Detecting contamination is harder than one might think. Literal deduplication can only catch exactly identical text; it cannot catch rewriting: a question may have words replaced, word order changed, or be translated into another language and back, with zero string overlap, yet semantically it is the same question. Conversely, an unusually high score alone cannot prove memorization—it could be genuine ability, or it could simply be that the answers were memorized. These two points together mean that neither “it has been deduplicated” nor “the score is suspicious” alone can establish contamination; more structured evidence is needed.

Defenses therefore need to be layered. First, construct new questions with time-based validity: if a question is released after the model's training cutoff date, it cannot possibly be in the pre-training corpus. Second, keep a set of private tests: model authors cannot access the question content, cutting off leakage channels. Third, screen public question sets using semantic nearest neighbors: retrieve semantically similar text for each question and check for suspicious solution traces—for example, a fragment like “the answer to this question is C” appearing in the corpus. Fourth, completely separate the development set used for hyperparameter tuning from the final gate set: the development set can be examined repeatedly, while the gate set is used only once before release. Fifth, record the number of times test results are viewed: the view count itself is a monitoring metric for overfitting—the more views, the closer the gate set becomes to the development set. For question sets that saturate quickly, they should also be replaced periodically rather than permanently using an old set that the entire industry has been grinding on for years.

This governance also has clear boundaries: private is not automatically representative. If the refund questions are all written by internal experts in standardized phrasing, then real users' colloquial language, typos, missing information, and roundabout expressions are all absent from the question distribution; no matter how well the model performs on a private set, after deployment it may still encounter ways of speaking it has never seen. Private sets only reduce leakage risk; they do not guarantee coverage of the real user distribution.

Write contamination governance as input and output: the inputs are questions, answers, mirrors, translations, explanations, and semantic nearest neighbors; the output is a contamination risk judgment and a decision to keep, replace, or downweight. Never being explicitly fine-tuned does not mean pre-training has not seen it, and literal deduplication cannot catch rewriting; only time-based new questions, private gate sets, and semantic screening working together can bring contamination risk into a controllable range.

# Automated Metrics, Large Language Model (LLM) Judges, and Human Blind Evaluation

The scorer outputs a number. Why should the scorer itself be evaluated? Because no scoring instrument is perfectly transparent; each has its own blind spots. If the scorer is not validated, the scores the model receives will be contaminated by the scorer's bias. To design a credible scoring system, first understand what each instrument is suitable for, what its main biases are, and how to calibrate them.

Exact match is the cheapest and most reproducible, but it wrongly penalizes equivalent expressions: if the model says "should refund 35 yuan" and "the refund amount is 35 yuan" mean the same thing, character comparison judges one correct and one wrong. Programmatic tests are strong on code and structural problems, can directly run assertions to check executable results, but they only cover the written assertions—correct or incorrect things outside the assertions are invisible. An LLM judge can understand semantics, making it suitable for evaluating attributes without standard answers such as helpfulness, completeness, and style, but it has built-in biases: it prefers answers that appear first or are longer, is more lenient toward text similar to its own style, is friendlier to outputs from its own model family, and can be led astray by prompt injection instructions in the output content. Domain human evaluation is closest to real judgment, suitable for high-risk and novel failure modes, but humans get tired, standards drift, and two annotators may give different conclusions on the same answer.

Calibration methods should correspond one-to-one with biases. The weakness of deterministic rules and programmatic tests is insufficient coverage, which can be supplemented with mutation testing and counterexamples: deliberately inject known-wrong amounts and fabricated reference IDs to see whether the rules catch them. An LLM judge's order bias is eliminated by swapping positions—evaluate the same pair of answers in both orders once; if the conclusion flips with the order, the judge is biased. Length and style biases are calibrated through blinding and human gold-standard calibration—anonymize the answers, align lengths, and use samples with existing human conclusions to check whether the judge keeps up with human judgment. Disagreement in domain human evaluation is resolved through double annotation and arbitration: the same answer is independently annotated by two people, disagreements are submitted to experts for arbitration, and annotation quality is monitored with consistency statistics.

The scoring combination for the refund scenario can be built like this: First, use programmatic checks for amounts, permissions, and reference IDs—these are executable, reproducible hard constraints; then have a blinded judge evaluate explanation clarity, handling language quality beyond the hard constraints; finally, for high-risk samples and samples where the judge and program disagree, hand them over to experts for arbitration. The reason for layering is that each instrument can only see its own dimension: the program cannot see whether the expression is good, the judge is unreliable on whether the amount is correct, and humans are too expensive and too slow to use on every question.

To understand the meaning of this combination, one intuition needs correcting: Putting multiple instruments together is not to vote and manufacture ground truth, but to make each one's blind spots visible. When the program catches an amount error that the judge missed, it shows the judge is not trustworthy on numbers; when the judge sees expression confusion that the program cannot judge, it shows the rules have insufficient coverage; when experts intervene on disagreements between the two, it shows the disagreement itself is a signal.

Written as input and output, the scoring combination takes deterministic rules, executable tests, an LLM judge, and domain human evaluation as input, and outputs scores for each attribute, disagreement records, and arbitration results. Rules are reproducible but have limited coverage; an LLM judge understands semantics but has order, length, and same-source biases; humans are qualified but get tired. Blinding, order swapping, double annotation, and gold-standard calibration together keep these biases within a visible range. Each instrument participates with its own error, but after combination, error can no longer slip through unnoticed.

# From Leaderboards to Release Gates

After a model's overall score improves, how do you turn evidence into a decision about whether to switch models? Leaderboards only answer “who is stronger”; release decisions must answer “will switching to it cause problems, is it worth it, and if something goes wrong, how do we roll it back.” These three questions each have their own gates, and the order cannot be reversed.

The first step is to define hard gates that must not regress. Dimensions such as the rate of unauthorized actions, privacy leakage, and critical language slices do not accept compensation from the overall score: even if the new model is ten points higher overall, as long as the unauthorized refund rate rises, or a critical language clearly degrades, it cannot replace the old model wholesale. Hard gates come first because once these risks occur, they cannot be hedged by average performance.

Only after passing the hard gates does the benefit comparison come: task quality, cost, p95 latency, and throughput must be considered together. Quality differences should be expressed as an interval from paired bootstrap rather than a single point estimate; also list the failure migration list—which errors from the old model were fixed and which new errors the new model introduced. Fixing ten old errors and adding eight new ones tells you more about the real consequences of switching models than a net difference of two points. No matter how complete offline comparison is, it only becomes release evidence after shadow deployment or small-traffic validation on representative real traffic. Shadow mode lets the new model observe real requests through a side channel, while small-traffic rollout routes only a small subset of users; both limit the impact radius of errors.

The carriers of evidence are the evaluation card and the results page. The evaluation card freezes the data version, configuration, and primary metrics to ensure that release review sees the same reproducible experiment; the results page reports overall results, slices, intervals, cost, and error examples together—missing any one of them leaves reviewers only a cropped number. The release policy must specify rollback thresholds: which line an online metric falls below triggers automatic or manual rollback, rather than leaving only a leaderboard. New failures exposed in production must flow back into the candidate set and, after human review, enter the next development cycle—these newly flowed-back questions must never be added directly to the current gating set, otherwise the judgment basis for this release is contaminated by post-release findings.

The inputs to a release decision are the interval for overall and slice differences, cost, latency, error migration, and online validation; the outputs are a plan to replace, route, retain, or roll back. An easily overlooked conclusion is that a “best model” often does not exist. One model may win in low-latency scenarios, another may be more stable in high-risk safety scenarios, and a third may excel at long context; when each of the three dimensions has its own winner, model routing is more appropriate than wholesale replacement—route requests by scenario to the model that is best at each, rather than forcing one model to win on all dimensions.

The last rule leaves no room: gating set failures cannot be cleared by deleting questions or changing metrics. Deleting questions the model cannot pass, or switching to a metric that happens to be more presentable, both lower the release gate itself. The value of a gate lies precisely in its immutability; otherwise evaluation turns from evidence into ritual.

# Connecting the Causal Chain

Evaluation runs from “Is this question worth testing?” all the way to “Can we release now?”. Every step builds on the previous one; if any link is missing, the final decision has no basis.

The first step is to define the target task distribution. The object of evaluation is never an abstract capability, but performance in a specific scenario: a refund assistant must be tested on policy Q&A, evidence citation, amount calculation, refusal to exceed authority, and multi-turn clarification, not on an overall ranking on a generic leaderboard. Only when the distribution is clearly defined do all subsequent steps have a common target.

The second step is to freeze data and inference conditions. The question version, few-shot, system prompt, temperature, token, number of candidates, retry rules, and scorer are all aligned and recorded as a snapshot; hosted models must also record the date and version. Only when conditions are frozen can the score difference between two models be attributed to the model itself, rather than to one run using single generation and another using the best of a hundred attempts.

The third step is to choose complementary scorers. Programmatic rules check amounts, permissions, and citation IDs; blinded judges evaluate explanation quality; experts arbitrate high-risk and disagreement samples. Putting multiple instruments together is not voting, but making each other’s blind spots visible.

The fourth step is to run and retain per-question results. The total score is temporarily set aside; per-question correctness is the raw material for all subsequent analysis: paired differences, slice performance, and failure migration are all computed from per-question records. Data that only keeps the total score is equivalent to discarding information at the source.

The fifth step is to estimate the difference and uncertainty. Use standard error and confidence intervals to assess whether a four-point gap on one hundred questions is credible, and use paired bootstrap to give an interval for the difference. Only a conclusion with an interval allows people to distinguish between “genuinely stronger” and “sampling fluctuation.”

The sixth step is to check slices, contamination, and cost before making a decision. The unauthorized-refund slice can veto an overall advantage; contamination screening determines whether the question set is trustworthy; the quality-cost curve determines whether this gap is worth the money. Offline wins must also pass shadow or small-traffic validation; a gating failure cannot be released by deleting questions or changing metrics.

After the six steps are completed, the output is no longer a score, but an executable decision: replace, route, retain, or roll back. The value of the entire chain lies in that every step can be independently checked, so the final conclusion can also be independently overturned or confirmed.