Skip to content
AI Knowledge MapSearch / Text directory

How do I check whether AI's citations exist and support the associated claims?

E006 · E Evaluating and correcting · Facts, evidence and uncertainty

Key answer

Check citations twice: establish that the work exists, then that its text supports the particular sentence. Professional-looking authors, journals and DOIs are unverified bibliographic information.

Full explanation

Walters and Wilder generated 84 short reviews on 42 topics with GPT-3.5/GPT-4 and checked 636 references, finding fabricated works and errors in real citations. These 2023 model/method results identify error types to check. Study abstract ALCE evaluated answer correctness separately from citation support: fluent answers and real papers can still have insufficient support. Sections 3 and 5

Check in order:

1. Existence: search titles, authors or DOIs on publisher sites/databases; confirm title, author, year and venue match. Broken links are not proof of fabrication; similar titles are not confirmation.

2. Correspondence: read the supporting passage, table or result with context. Split multifactual sentences into claims; narrow claims or add sources when only partly supported.

3. Evidence type: distinguish experimental results, discussion hypotheses and another study quoted by the source. Trace key secondhand claims to originals.

4. Record status: supported, partly supported, unsupported or unverified, with locations and issues.

Verify each key claim with original sources, supporting locations and scope. Disclose abstract-only access; do not present search snippets as full-text evidence. Leave missing citations unverified rather than inventing similar works.

This makes checks explicit; personally open important sources before delivery, especially for numbers, causality and populations. More citations do not ensure a complete evidence chain.

References

1. Walters and Wilder (2023), Fabrication and errors in the bibliographic citations generated by ChatGPT, Scientific Reports, abstract

2. Gao et al. (2023), Enabling Large Language Models to Generate Text with Citations, EMNLP, Sections 3 and 5–6