What to verify after an AI agent says tests pass

Green CI from an agent is not a merge signal. It is a claim: "I ran something and it exited zero." Your job is to check whether that claim covers the bug, the intent, and the failure modes you care about. I treat "tests pass" as the start of a short verification sequence — not the end of review. 1. Inspect the diff, not the summary Open the file list first. Ignore the agent's narrative until you can answer: Which files actually changed? Do they match the human intent in one sentence? Any lockfiles, renames, config, or test fixtures you did not ask for? Agents often pad the suite while touching unrelated helpers. If the diff is wider than the ticket, pause before you trust the green check. A passing suite on the wrong surface is still the wrong merge. 2. Read the assertions (adversarially) Open the new or edited tests and ask: would this fail if the original bug came back? Watch for: Asserts that only check "something returned" or status 200 Snapshots that absorb any behavior change Happy-path-only coverage with no edge or negative case Tests that fail solely if the function is deleted If the assertion would still pass with the regression restored, the suite is theater. Request a tighter assert before you approve. Prefer one sharp negative case over five soft positives. 3. Exercise the failure paths Green tests often skip the paths that hurt in production: Invalid input, missing auth, permission denied Empty collections, timeouts, partial writes Feature-flag off, second call, idempotency Pick the failure mode closest to the ticket and ask whether any test forces it. If not, either add that case or manually exercise it before merge. Agents optimize for "looks covered." You optimize for "breaks when broken." 4. Reproduce the command locally Do not trust the agent's pasted output alone. Run the same command on your machine (or the same CI job) with the PR branch checked out: # example — use whatever your repo actually runs npm test -- path/to/relevant.spec.ts Check: Same command the agent claimed to run? Same working directory / env assumptions? Flakes, skips, or "passed with warnings" you would not accept? If you cannot reproduce green locally, you do not have a pass — you have a story. Fix the story before merge. Sequence, not vibes Order matters: diff → assertions → failure paths → local reproduce. Skip ahead and you rubber-stamp confidence. Stop early when the file list or asserts are weak do not sink twenty minutes into a suite that never could catch the bug. This is the same bar I use on agent PRs elsewhere: keep the speed, keep your judgment. Green is necessary. It is not sufficient. If you want the packaged checklist, Cursor-oriented rules, and review prompts I use on agent PRs, the AI Agent Code Review Kit is here: https://chopragunji.gumroad.com/l/nxoboi — Riven Desk What do you check first after an agent claims tests pass — file list, asserts, or a local re-run? Drop your sequence in the comments.