Key answer
Fix tasks and scoring criteria, then change only the feature under study where possible. One generated winner does not represent long-run performance.
Full explanation
Sclar et al. found semantically irrelevant delimiters, spaces and capitalization substantially affected some models on few-shot tasks, with inconsistent rankings across models. Sections 3–4 A role version also adding background, examples and detail cannot isolate a role's effect.
Keep at least these conditions consistent:
· Questions/material: same cases, file versions and source scope.
· Model/environment: same version, tools and generation settings where possible; record dates and avoid unequal search access.
· Context/investment: equivalent conversation history, examples and output requirements. Match attempts/cost for method comparisons or report additional effort when comparing full processes.
· Evaluation: define factual accuracy, format and usability before seeing results; score separately.
Start with paired real cases, both methods answering each question, repeated generations and records of successes, errors and time. Include ordinary and important exceptional cases; selecting only favorite outputs hides failure frequency. For writing, hide method names and swap display order to reduce preconceptions.
Compare prompts A and B on the same customer-service questions with identical material/settings. Run both per question. Score predefined factual accuracy, rule compliance and usability; retain all outputs and timing, not only successes.
Small trials reveal obvious differences without statistical certainty. If differences resemble noise or flip with a few cases, add representative examples. For similar quality, choose lower effort/easier maintenance and revalidate on new models/tasks.