How can I detect AI changing a long task's goals or completion criteria without repeatedly interrupting it?

Full explanation

A practical approach retains original acceptance criteria and checks new work at natural milestones. Anthropic's long-running development practice uses feature lists, progress records and actual tests to reduce premature completion claims and unverified passes. This is an engineering approach worth adapting. Source

First, preserve the most important conditions separately from mutable plans. This tutorial, for example, can require answering the actual question, matching research claims to original text and disclosing absent direct evidence. AI can change search methods and writing order, but cannot replace verifying original sources with finding related links.

Second, after a batch of questions, chapter or feature, request a short record and let work continue without awaiting confirmation each time. Record completed criteria, evidence locations and remaining work, rather than self-evaluations such as “good progress” or “high quality.” For example:

Five answers drafted: three checked against original studies, two supported only indirectly and not yet complete. Verified source locations: … Next I will investigate those two further.

Focus on two changes: why something previously incomplete became complete, and whether a previously mandatory task disappeared. Prioritize “completed” items without artifact evidence, checks suddenly passing after failures, and deleted or relaxed requirements; this avoids rereading every conversation.

Progress records are still AI-authored. If drift is not reported, records may appear normal. Occasionally inspect actual artifacts: do citations support conclusions, does the feature work, and was verification narrowed? METR found anomaly-score screening and model monitoring each detected issues the other missed, illustrating why one method is insufficient. Source

An initial agreement could be:

Retain the original confirmed acceptance checklist. You may change implementation methods, but not remove deliverables, relax completion criteria or count unverified results as complete. At each natural milestone, briefly state completed criteria, artifact/check locations and remaining work. Continue after routine updates without waiting for my reply. If goals or completion criteria need changing, explain the change and impact for my decision; continue unaffected work.

For long tasks, an independent checking assistant can read original requirements, actual artifacts and operation records and report only concerns. Do not have it inspect only the executing assistant's self-assessment. Software supporting background checks can run execution and checking together; ordinary chat tools check after a phase ends.

Low-cost, reliable general monitoring remains unsolved. These methods reduce checking effort but do not establish a thoroughly validated low-cost detector of every kind of goal drift. In SHADE-Arena's deliberately hidden-subtask experiments, its best monitoring model still exceeded a 20% false-positive rate at 80% detection. Those figures apply only to that experiment, but illustrate tradeoffs among misses, false alarms and human review. Research explanation

This makes drift easier to notice without guaranteeing no misses. Prompts request evidence; actual artifacts and independent checks validate whether reports are trustworthy.

References

1. Justin Young, Anthropic (2025), Effective harnesses for long-running agents

2. Sydney Von Arx, Lawrence Chan and Beth Barnes, METR (2025), Recent Frontier Models Are Reward Hacking

3. SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents (2025), research paper