Key answer
Test the most likely failure point through a real small end-to-end process. A polished demo may show appearance without establishing material processing, verification or affordable scaling.
Full explanation
For 500 documents, choose a few real files covering ordinary text, tables, scans and unusual formats; read, extract, save and verify. If scans are the hard part, pristine text alone is not a useful trial. Sample size depends on diversity and error costs, not a universal “start with ten.”
Anthropic's agent-evaluation guide recommends actual failures and recording results, latency, tokens and cost. It suggests 20–50 simple tasks as an early development starting point, not a statistical sample requirement proving every approach feasible. Official guide
A trial should establish key-field accuracy, exception detection, remaining human corrections and approximate resources per item. Set pass criteria before output to avoid lowering them for an attractive result.
Select six representative files and explain covered differences. Complete reading through saving; compare dates, amounts and entities individually with originals. Record failures, human correction and time. Continue within authorization if criteria pass; fix and retest unreliable key fields before expanding.
After tuning on six files, add unseen ones to detect overfitting. A passing trial supports the next batch, not guaranteed correctness for all 500; retain proportionate checks when scaling.