Verify the Claim
Checks what the agent said it did against what the diff actually does.
When to reach for it
Every time a summary arrives saying the work is complete and the tests pass — especially when it is long, confident, and pleasant to read.
What changes
- Each sentence of the summary becomes a numbered claim marked verified, false, or unverifiable, with the evidence named.
- Files changed but not mentioned in the summary are listed explicitly — that is where the surprises live.
- "Tests pass" only counts with command output produced in this session, not recalled from an earlier one.
- Catches the specific failure where a claimed behavior is a TODO, an early return, or an empty catch block.
Pairs with
- Spot the SlopNames the specific tells of generic machine-made code and design, so you can delete them.
- Accessibility PassCatches the accessibility failures generated interfaces produce almost every time.
- Comment AuditStrips the comments that repeat the code and keeps the ones carrying facts the code cannot state.
Verify the Claim
A summary is a description of intent. The diff is what happened. Check one against the other.
1. Turn the summary into numbered claims
Split it into individual assertions, one per verb: added, fixed, removed, renamed, handles, tested, verified, ensures. A four-sentence summary usually contains eight claims.
2. Name the evidence each claim needs
For every claim, state what would prove it: a specific hunk in the diff, a passing test with visible output, a command result, a file that now exists. A claim with no possible evidence is not a claim.
3. Read the whole diff, including unmentioned files
Read it in full, not just the parts the summary points at. List every file changed but not mentioned. Deletions and config edits hide here.
4. Look for the four common gaps
- Claimed but absent — the summary describes a change the diff does not contain.
- Present but unclaimed — the diff contains a change the summary does not mention.
- Claimed as tested, never run — the test exists but the suite was not executed, or was executed before the last edit.
- Claimed as handled, structurally unhandled — the branch exists but returns early, contains a TODO, catches and ignores, or logs and continues as though nothing happened.
5. Re-run anything quoted
If output was pasted, run the command again now and compare. Output from three edits ago describes a codebase that no longer exists.
6. Report a verdict per claim
Verified, false, or unverifiable — each with its evidence or the reason none could be found. Then decide what to do about the false ones.
Rules
- Do not accept "tests pass" without the command and its output in this session.
- Do not treat the summary as evidence about the diff. It is evidence about what was intended.
- Do not skip files the summary did not mention. That omission is the single most reliable signal worth following.
- Do not mark a claim verified because it is plausible, or because the code near it looks correct.
- Do not stop at the first false claim. The rest of the list is still worth checking, and false claims cluster.