Model Selection and Cost: Finding the Minimal Viable Solution That Satisfies Constraints

Questions like “which model is best” come up repeatedly in model selection discussions, but it is not an answerable question: it lacks an objective function. A model can only be compared after the task is specified and what counts as “good” is defined; leaderboard rankings detached from the task do not provide a basis for model selection. The same leaderboard conclusion has no shared explanatory power across three types of tasks—invoice extraction, marketing copywriting, and refund approval—because these tasks differ completely in how they define “correct,” in the cost of errors, in required context, in output structure, and in permission requirements. Invoice extraction cares about whether fields are complete and accurate; copywriting cares about whether the expression is clear and effective, and errors are usually tolerable and easy to correct; refund approval is a high-risk decision with permissions, where a single mistaken “approve” can directly cause a financial loss. Measuring them with the same score is equivalent to assuming that errors across these three task types can be converted into one another, when in fact they are not even in the same unit of measure.

Therefore, the first step in model selection is not to find a model, but first to pin down the task conditions. It is necessary to explicitly write out: input distribution—where requests come from and what they look like; success event—what output counts as a success; high-risk slices—which subsets have an extremely high error cost and must be measured separately; latency SLO—for example, a p95 latency upper bound; data boundary—where data is allowed to be stored and whether it can cross a specified region; throughput and budget—peak request volume and total affordable cost. Only after these conditions are fixed do the scores obtained by different models become comparable evidence; when conditions are inconsistent, differences in scores cannot be attributed to the model itself being better or worse.

Among these conditions, some are hard constraints and some are optimizable objectives. Taking a refund assistant as an example, the hard thresholds might be: unauthorized operation rate ≤ 0.5% (the proportion of unauthorized operations must be kept very low), quality issue eligibility accuracy ≥ 90% (accuracy when determining whether it qualifies for a refund), p95 latency ≤ 1.8 s, and data must not leave the designated region. These thresholds do not allow “partial satisfaction”: a model with a 0.7% unauthorized operation rate cannot use a lower price to “make up for” the unauthorized operation risk, no matter how much cheaper each call is. A hard constraint failure is failure; a lower price or a higher score on ordinary samples cannot offset it. In contrast, “explanation clarity” and “cost” are optimizable objectives—among candidates that satisfy all hard constraints, the clearer the explanation and the lower the cost, the better, but they can never be traded against hard constraints.

Thus the model selection problem can be stated precisely as: within the constraint set C, which “model—prompt—tool” configuration has the lowest long-term total cost? Note that the decision object here is the entire configuration, not the model name. The model name is only part of the solution: the same model paired with different prompts and different tool integration methods can produce completely different constraint satisfaction and costs. Calling model selection “which model to choose” is an oversimplification; the real output of model selection is a set of candidate configurations.

This gives the inputs and outputs of the model selection process. The inputs are the six categories of conditions: task distribution, success definition, risk slices, SLO, data boundary, and budget; the output is the set of candidate configurations that satisfy all hard constraints. The order of the process cannot be reversed: first define “success” and the non-compensable hard thresholds, then compare candidates within those thresholds. Leaderboard scores are interpretable only when all these conditions are the same—when two models are evaluated under the same input distribution, the same success definition, and the same risk slices, differences in scores reflect real capability differences. Conversely, any hard constraint failure cannot be offset by a lower price or a higher score on ordinary samples.

When prototype performance falls short, the most common confusion is whether the model is not strong enough, or whether retrieval, prompts, or the process itself is flawed. These two diagnoses point to completely different directions for fixing the issue: switch to a stronger model, or fix the data, prompts, and pipeline. The way to separate them is to first use a strong candidate to establish the system's upper bound.

The specific approach: choose a candidate with strong capability, sufficient context, and stable tool integration, run it on a representative sample, and give it complete evidence—that is, the 'correct' context and tool results in the oracle sense. The score measured in this run is the achievable upper bound for the current architecture (retrieval method, prompt structure, tool chain, acceptance criteria) when model capability is sufficient. Then examine the results in two cases.

If the strong model still fails under oracle evidence, it means the bottleneck is not the model but the task itself: task definition, data quality, tools, or acceptance criteria have problems, and you should fix these first rather than continue changing models. If the strong model passes, it means the current architecture itself can do the task well, the upper bound holds, and then you can start ablation: swap in smaller models one by one, reduce context, reduce the reasoning budget, and measure the resulting loss each time something is saved. The principle of ablation is to hold the remaining variables constant and replace only one model or one reasoning configuration at a time; otherwise changes in performance cannot be attributed. The entire comparison process must freeze the evaluation set and the scorer; you cannot change questions while comparing—once the questions and scoring criteria change, scores measured earlier and later are no longer comparable.

The value of the strong model baseline is also reflected in error analysis. Retain the per-sample comparison differences and examine error migration between old and new configurations: which samples were originally correct but wrong after the configuration change, and which were originally wrong but correct after the change. The error migration map can show on which types of inputs the cost of each 'saving' specifically falls. At the same time, evaluation must cover the end-to-end fallback chain, not just measure the accuracy of a single output—if one step is correct but the whole chain is wrong, it is still a failure in a real system.

It should be emphasized that the strong model baseline is not the default production solution. It is only a diagnostic reference: the upper bound tells you how good this architecture can be at most and what the per-sample error distribution is. Privacy requirements, latency budgets, or licensing restrictions may prevent this strong model from being deployed from the start; the purpose of the upper bound is to calibrate the improvement space and locate bottlenecks, not to automatically become a production recommendation. The baseline's input is a frozen evaluation set, sufficient evidence, and stable tools; its output is the achievable quality upper bound of the current architecture and per-sample errors. When it fails, fix the task, data, or tools first; when it passes, replace variables one by one to perform ablation.

When putting quality, risk, latency, and cost into the same decision, the easiest mistake is to arbitrarily weight them into a single total score. Where do the weights come from? Who decides that 1 quality point equals how many yuan of cost? Once a recognized conversion rate is lacking, any weighted total score is an artificial construct, and the ranking result will reverse as the weights are fine-tuned. Model selection should be expressed as constrained multi-objective optimization: minimize TotalCost(m) over candidate configurations m (i.e., “model—prompt—tool” configs), while satisfying a set of hard constraints—Qualityₖ(m) ≥ Qₖ (the k-th key quality is not lower than the lower bound Qₖ), Riskⱼ(m) ≤ Rⱼ (the j-th risk is not higher than the upper bound Rⱼ), P95(m) ≤ L (p95 latency does not exceed upper bound L), Privacy(m) ∈ C (privacy category falls within the allowed set C). Quality, risk, latency, and privacy enter the problem in the form of thresholds; first execute these hard constraints to filter out the feasible region, then compare cost, throughput, and maintenance burden within the feasible region. Candidates that fail thresholds are directly eliminated, no matter how outstanding they are on other dimensions.

For thresholds to be truly executable, each dimension needs a measurable expression and must clarify what it cannot be offset by. Critical quality is expressed as slice pass rate and confidence interval—the pass rate and its uncertainty on high-risk slices are the evidence, not offset by average improvement on ordinary samples. Safety and permissions are expressed as event rate and severity—the frequency and consequence level of events such as unauthorized access and leakage cannot be offset by more natural writing style. Latency is expressed as TTFT, p95/p99, and timeouts—tail latency and timeouts determine user experience, not offset by average latency. Cost is expressed as cost per successful task and peak capacity—how much is actually paid for a successful task and whether it can hold up at peak, not offset by token unit price. Governance is expressed as region, retention, audit, and exit capability—whether data can remain in the specified region, whether retention policies are compliant, whether audit is possible, whether exit is possible, not offset by model capability score.

Within the feasible region, if there is no recognized conversion rate among multiple objectives, use the Pareto frontier for further screening. A candidate is dominated if there exists another candidate that is no worse in all dimensions and strictly better in at least one; the Pareto frontier is all non-dominated candidates in the feasible region. Being on the frontier does not mean automatic victory: it only indicates that this candidate is not comprehensively dominated and is worth further discussion combining business weights; the final trade-off still requires the business side to take a position among quality, risk, latency, and cost, while the frontier shrinks the “candidates worth serious consideration” from the full set to a small set.

The input to the entire process is the candidate set m, each quality Qualityₖ, each risk Riskⱼ, p95 latency, privacy category Privacy, and the corresponding thresholds Qₖ, Rⱼ, L, C; the output is the solution with the lowest total cost TotalCost in the feasible region. The order is fixed: first execute hard thresholds to filter out the feasible region, then look at Pareto frontier and cost. Being on the frontier only means not comprehensively dominated, not automatic victory.

If one model's per-call cost is only a quarter of another's, is using it really cheaper? The price list only describes the marginal price of a single call, but a complete request flow for a real task includes many steps: initial answer, retrieval, retries after failure, escalation to a stronger model, tool calls, manual review, and failure remediation. After switching to a cheaper model, the usage of these steps changes—more initial-answer failures, more retries and escalations, longer manual review queues. There are also fixed costs: deployment redundancy, monitoring, evaluation, supplier integration, and migration. Attribute all of these to mature business results, not just to the unit price on the price list.

The core formula for full cost is: full cost per successful task CostPerSuccess = (Ccall + Ctool + Cretry + Chuman + Cinfra + Closs) / Nsuccess. The numerator aggregates resources and losses across the entire task pipeline: Ccall is the model call cost, Ctool is the retrieval and tool cost, Cretry is the cost of retries and escalation to a stronger model, Chuman is the manual review cost, Cinfra is the infrastructure cost, Closs is the loss caused by failures; the denominator Nsuccess is the number of genuinely successful tasks independently confirmed, not the number of tasks the system itself claims to have completed.

The choice of denominator is the most error-prone part of the formula. If you only count tasks the system claims to have completed, you miss two types of outcomes: users who give up and wrong automatic approvals. A user leaves before getting a result, but the system records it as “completed”; a task with wrong automatic approval is treated as success by the system, while the business side has already suffered a loss. Success must be defined by business state independent of the system, and the denominator counts only real business success, so that costs fall on the actual results produced.

The failure loss Closs in the numerator may not be precisely priceable. The expected loss from wrongly approving a refund, the reputational loss from a single unauthorized action, often cannot be converted into a reliable monetary amount; for such high-risk scenarios, using a hard threshold is more honest than forcing a price. But even if failure loss is not monetized, the accounting should at least show labor and retries: if they are hidden, comparisons reward solutions that shift costs to the operations team—the model is cheaper, but overtime increases, and total cost is actually higher. This is the mechanism of “a smaller model is cheaper per call but more expensive in the end”: a low per-call price brings more retries, escalations, labor, and failures; these increments all go into the numerator, while the denominator shrinks due to lower success rate, so CostPerSuccess rises. High-risk losses that cannot be priced should be kept as hard thresholds rather than converted into the cost formula.

Run the previous formulas through a concrete scenario: process 1000 refund tasks. Candidate S costs ¥0.02 per call, candidate M ¥0.08 per call, and candidate L ¥0.18 per call (¥180 for 1000 calls). Looking only at the price list, S is obviously the cheapest; looking at the full cost per successful task, the conclusion is the opposite. The comparison has two steps: first pass the hard thresholds, then perform the full economic accounting. A cheap candidate whose over-authorization rate exceeds the limit cannot enter the cost track at all and is eliminated directly; after entering the track, compare by CostPerSuccess.

The accounting process is shown in the table below. In each row, add the call cost, manual review (¥1 each), and failure remediation (¥3 each) to get the numerator, then use the actual number of successful tasks as the denominator:

S's failure mechanism is clear at a glance: calls cost only ¥20, but a 78% success rate means many tasks require manual work and remediation—¥180 manual plus ¥120 remediation, for a total expenditure of ¥320 spread over 780 successful tasks, or ¥0.41 per successful task. M's call cost is four times S's (¥80), but its success rate is higher and manual work and remediation are lower, so the total expenditure of ¥151 spread over 900 successful tasks is about ¥0.168. L has the strongest capability and also the highest call cost (¥180), with a 93% success rate and further reductions in manual work and remediation, so the total expenditure of ¥218 spread over 930 successful tasks is about ¥0.234. Under the given assumptions, M wins: it is neither the cheapest to call nor the most capable, but it has the lowest cost per successful task.

This conclusion is sensitive to assumptions. If the manual unit price rises from ¥1, or the failure remediation unit price changes, the relative ranking of M and L will shift; if the traffic structure changes—for example, the proportion of difficult problems in the input distribution rises and S's success rate falls further—the conclusion likewise changes. Therefore, the accounting is not a one-time task: after obtaining the cost per successful task, you must perform sensitivity analysis on the manual unit price and traffic structure to confirm that the winner still holds under parameter perturbations. The input to the case is 1000 refunds and the threshold, call, success, manual, and remediation data for S, M, and L; the output is the feasible candidates and their cost per successful task—first eliminate those that fail on over-authorization or latency, then calculate and compare using the formula, and finally test the stability of the conclusion with sensitivity analysis.

When the output quality of two endpoints is similar on the evaluation set, the decision turns to non-behavioral dimensions such as contracts, versions, and exit capability. Items that need to be compared item by item include: whether data is used for training, data retention period, data region, encryption method, audit capability, model version locking, rate limits, SLA, batch pricing, content policy, and service termination terms. Self-hosting also needs to additionally account for GPU utilization, patch maintenance, on-call duty, and disaster recovery costs—what is saved is pay-per-use, what is added is the entire operations surface. Open weights provide control, but control does not automatically bring licensing, data provenance, or security guarantees: being able to modify a model does not mean you have the right to commercial use, nor does it mean the training data and dependent components are clean and auditable.

Vendor assessment can be carried out item by item according to 'question—evidence'. To determine whether the version can be locked, the evidence is snapshot identifiers, change notifications, and regression windows—silent model updates can quietly invalidate a configuration that has already passed evaluation, so you must know which snapshot is running in production, whether there was notification before the change, and whether there is a regression window after the change. For data destination, the evidence is contracts, regions, retention terms, and the list of sub-processors. For capacity assurance, the evidence is quotas, burst policies, SLA, and historical failure records—even after signing an SLA, you must also check whether it is sufficient to cover peaks and bursts. For exit cost, the evidence is actual testing of alternative endpoints, migration scripts, and recorded compatibility differences.

Exit capability cannot rely on assumptions; it must rely on drills. Conduct an exit drill: export prompts and evaluation sets, switch to a backup endpoint, verify output contracts and tool compatibility, and measure actual recovery time. The 'equivalent model' claimed by a vendor may differ in format, refusal behavior, and tokenization; the abstract interface layer can only reduce migration costs, not eliminate behavioral differences. Having the same interface does not guarantee the same format, refusal behavior, and tokenization; the time actually taken to switch and restore service is the most credible evidence of exit capability.

The inputs to vendor assessment are model behavior, contracts, regions, retention periods, versions, capacity, licensing, and exit drill results; the outputs are governable operating plans and alternative paths. Similar quality is only a necessary condition; what ultimately determines the final outcome is often these governance dimensions.

"Choosing one model" is often not the final system design, because different steps in an application have completely different difficulty and risk: intent classification, query rewriting, core reasoning, code execution, and safety review each have different requirements for capability, latency, and trustworthiness. Having a single large model handle all steps means using the same level of capability to deal with all levels of risk, which is both expensive and difficult to verify step by step. The alternative is to statically divide responsibilities by component: deterministic rules check permissions first, a small model extracts order fields, a medium model interprets policy, and high-risk exceptions are routed to humans. Each component is independently responsible and independently verified—permission checks are verified by the unauthorized access rate, extraction is verified by field accuracy, and high-risk judgment is verified by recall of cases routed to humans—the whole chain is more testable than a single large model and makes it easier to locate which component failed.

Dynamic routing is another approach: predict difficulty at the request level, then decide which model to send it to. It may save further cost compared with static division of labor, because simple requests no longer go through a strong model. But it introduces three new problems: erroneous downgrading—difficult requests are misjudged by the router as simple requests and sent to a small model; cascading latency—the routing judgment itself takes time, and with possible multi-level retries, tail latency deteriorates; router drift—the router model's own distribution changes with traffic, and routing decision quality degrades over time. Therefore dynamic routing must not be evaluated together with the underlying model: first establish a static baseline and confirm that each component's threshold is met, then evaluate routing as an independent system separately; the cost savings brought by routing belong to the router and cannot be credited as gains of the underlying model.

Tools are not a free add-on to model capabilities either. Within the same toolchain, different candidates differ in schema compliance, error recovery, and context utilization: a model that performs well on pure question answering may perform poorly on schema validation and retry strategies after tools are connected. Therefore any candidate with tools must be tested on the same end-to-end toolchain, not extrapolated from tool-free evaluation scores.

The inputs of static model composition are the difficulty, risk, structure, and tool requirements of each component; the output is the division of responsibilities among rules, small models, medium models, strong models, or humans. By fixing responsibility per component, permission checks, field extraction, and high-risk judgment can be verified separately; dynamic routing is a subsequent independent system built on top of the static baseline, and its benefits must be attributed separately.

When the preferred model times out, is rate-limited, has low confidence, or outputs invalid results, what does the user actually experience? This question must be answered at the selection stage, not left to be answered when a production incident occurs. Fallback design must define in advance the branches corresponding to each exception: retry the same endpoint, upgrade to a stronger model, switch to deterministic code, ask the user for additional information, return a cached result, transfer to a human, or fail explicitly. Each branch must have a budget—total tokens, total latency, and side effects. Pay particular attention to cascade contamination: content generated by the first response must not be passed to subsequent models without review; otherwise errors or injections in the first response will spread along the chain, and the upgrade branch itself will also be contaminated.

Evaluation of the fallback chain relies on fault injection: proactively create supplier 429 rate limiting, stream interruption, tool timeout, schema changes, and regional unavailability, and see how the system ultimately performs under these conditions. What needs to be measured is not whether the branches 'look reasonable,' but the final task success rate, p95 latency, number of repeated actions, and human load under fault conditions. A 99.9% available primary model does not mean the entire cascade is equally reliable—each dependency in the fallback chain is connected before or after the primary model, and if any one is unavailable, end-to-end availability is affected.

The availability of serial independent dependencies is approximately the product of the availability of each dependency: Asystem ≈ ∏ᵢ Aᵢ, where Aᵢ is the availability of the i-th dependency. Even when each component is very reliable, the product still declines level by level: for three serial dependencies each at 99.9%, end-to-end availability is only about 99.7%; for ten, it is only about 99%, and it drops below 99% only at the eleventh. This approximation explains the common phenomenon of 'components individually reliable, but end-to-end availability significantly lower.' It is only an approximation: in real systems, failures are often correlated—a regional outage may take down both the model and the tools simultaneously—so real correlated failures, repeated side effects, and total latency cannot be derived from the product; they must be directly verified through fault injection experiments.

The inputs to fallback design are the trigger conditions—timeout, rate limiting, low confidence, invalid output—and the availability Aᵢ of the i-th dependency; the outputs are the specific strategies for the branches—retry, upgrade, deterministic code, ask, human, or failure—and an estimate of system availability Asystem. The selection decision must include this branch table and its fault injection results: a candidate that performs well only under normal conditions, paired with an untested fallback chain, is equivalent to leaving the worst-case cost until after launch.

Today's optimal model may no longer be viable next month: price adjustments, model version updates, context length changes, supplier policy modifications, traffic language structure drift, task rule changes, and labor cost fluctuations—any one of these can overturn the previous conclusion. Model selection is not a one-time comparison at procurement time, but an ongoing process with triggers. Trigger conditions that must be preset include: model snapshot updates, price changes, pass-rate drift on critical risk slices, error budget exhaustion, issuance of new risk requirements, and backup endpoint failover failure. Once a trigger occurs, rerun the frozen evaluation set and real traffic sampling, rather than waiting until an incident has occurred and then evaluating ad hoc.

Every reassessment and every original decision must be recorded on a decision card, including: candidate list, exclusion reason for each candidate, data version, thresholds, confidence intervals, cost assumptions, approver and reassessment date, and exit plan. The value of the decision card is to prevent memory loss: a candidate was once eliminated due to privacy or unauthorized access, and months later when a new version leaderboard appears, the team may forget the original exclusion reason and want to reintroduce it based solely on leaderboard scores. The decision card preserves this causal chain; when reintroducing, one must first re-examine whether the original exclusion reason still holds.

Evaluation data itself must also be protected from contamination. Public leaderboards can only be used to generate candidate lists, not to replace a private representative evaluation set—the leaderboard composition is unrelated to your task distribution, and its scores are not interpretable when task conditions differ. Conversely, private sets have their own pitfall: repeatedly tuning parameters, changing prompts, and swapping models against the same private set will also overfit to that set, and evaluation scores no longer represent real traffic. Periodically mixing in real samples and updating the private set when necessary are standard means to combat contamination in both directions.

Inputs to lifecycle reassessment are changes in price, model version, task distribution, policy, labor cost, and error budget; the output is one of four decisions: continue use, re-compare, canary switch, or exit. Historical optimality holds only for the task distribution, prices, and assumptions at that time; it cannot be permanently inherited by inertia—reassessment is not to overturn old decisions, but to ensure that “current optimal” always corresponds to current conditions.

Connect the previous steps in causal order. From the question "which model is best" to verifiable production practice, model selection consists of six steps, each providing the premise for the next.

Step one: Define the task distribution and hard constraints. Write out the input distribution, success events, high-risk slices, latency SLO, data boundaries, and budget, and express the hard thresholds for quality and risk as measurable expressions. Without this step, none of the subsequent comparisons has a common coordinate system.

Step two: Use a strong model to establish an architectural upper bound. Run the strong candidate on a frozen evaluation set with sufficient evidence: if it still fails under oracle evidence, the bottleneck is in the task, data, or tools—go back and fix that first; if it passes, the current architecture is feasible and the upper bound holds.

Step three: Compare per-sample differences under fixed conditions. Replace only one variable at a time—model or inference configuration—while keeping all other conditions unchanged, and record the error migration between the old and new configurations: which samples newly fail, which are corrected, and which input types bear the cost.

Step four: Eliminate candidates that do not meet the thresholds. Use hard constraints to carve out the feasible region: unauthorized operation rate, slice pass rate, latency p95, privacy categories—any one failing means immediate elimination; low price and high scores on ordinary samples cannot offset threshold failures.

Step five: Calculate the full cost per successful task. For feasible candidates, compute using CostPerSuccess: the numerator is the full-chain expenditure on calls, tools, retry escalation, human labor, infrastructure, and failure losses; the denominator is the number of true successes confirmed by independent business state. When a low unit price brings more retries, human labor, and failures, the final cost is actually higher; if necessary, combine Pareto frontier and sensitivity analysis to determine the winner.

Step six: Roll out gradually, reevaluate according to triggers, and exit. The winner is first validated on a small traffic share, while recording the decision card and setting triggers for price, version, slice drift, error budget, etc.; when triggers fire, rerun the frozen set and real sampling to decide whether to continue use, re-compare, shift traffic, or exit. The historical best holds only for the data and assumptions at that time; model selection is a cyclical chain, not a one-time answer.