Skip to content
AI Knowledge MapSearch / Text directory

How do I check regressions when AI fixes one part but damages another?

E017 · E Evaluating and correcting · Error diagnosis and specific feedback

Key answer

Check both whether the original error disappeared and whether previously correct content remained correct. The latter is regression checking for code, documents, spreadsheets and rules alike.

Full explanation

CheckList organized sentiment, duplicate-question and reading-comprehension tests by capabilities and behaviors, exposing errors beyond aggregate accuracy, including changes that should not affect answers. Sections 2–3 Adapt this by listing invariant properties and applying the same checks to old/new versions.

Scale checks to impact:

1. Keep the earlier version for comparison and restoration.

2. Retest the failing case with its original input rather than an easier replacement.

3. Check adjacent rules/dependencies: timelines after date changes, totals after quantities, callers after function changes.

4. Recheck key invariants: names, numbers, citations and unauthorized rewrites; spreadsheet formulas, units and summaries; relevant software tests and important old cases.

Changing “three days” to “five business days” also affects summaries, diagrams, calculations and the definition of business days. Correct eligibility conditions should remain unchanged.

Correct the refund-review duration and every reference to it. Preserve eligibility and amount rules. Report actual changes, dependent content checked and unchecked scope, comparing with the previous version.

A typo may need only a diff; calculation rules need old, boundary and failing cases. Passing tests covers checked scope only. “Nothing else changed” is not a diff. If a regression appears, retain valid fixes and repair the new problem rather than regenerating everything.

References

1. Ribeiro et al. (2020), Beyond Accuracy: Behavioral Testing of NLP Models with CheckList, ACL, Sections 2–3