I let an AI agent write a README for a real repo — here's what it produced

Most small open-source projects have a README that stopped matching the code two years ago. I wanted to know whether an AI agent could write one that actually reflects what's in the repository — not a plausible-sounding one, but one where every flag and environment variable it mentions is real. So I built a small pipeline and pointed it at someone else's repository. The setup The rule I started with: the model never decides what's true about the code. The pipeline has four stages, and only one of them involves an LLM: Read — fetch the repository's files over HTTP. Extract facts — parse pyproject.toml, argparse calls, os.environ lookups, the LICENSE file. Deterministic, no model involved. Generate — hand the model the files and the extracted facts, ask for a README. Verify — check the output against the facts. Any CLI flag, environment variable, or license claim that isn't in the extracted facts is treated as invented, and the deliverable is rejected. Stage 4 is the part that makes this usable. A README that documents a --dry-run flag which doesn't exist is worse than no README at all, because someone will try it. The repository sloria/ped — a small MIT-licensed CLI tool that opens Python modules in your editor. A few hundred lines across a handful of source files. Small enough to reason about, real enough to be interesting. I picked it because it already has a decent README. That made it a fair test: could the agent produce something comparable, without copying it? What the agent read Nine files, chosen by a priority list — config first, then entry points, then the rest: LICENSE README.rst pyproject.toml src/ped/__init__.py src/ped/guess_module.py src/ped/install_completion.py src/ped/ped_bash_completion.sh src/ped/pypath.py src/ped/style.py What it extracted, deterministically { "name": "ped", "version": "3.0.0", "license": "MIT", "requires_python": ">=3.8", "scripts": ["ped"], "cli_flags": ["--complete", "--editor", "--help", "--info", "--version", "-e", "-h", "-i", "-v"], "env_vars": ["EDITOR", "NO_COLOR", "PED_EDITOR", "PED_OPEN_DIRECTORIES", "SHELL", "VISUAL"] } Two of these were harder to get than they look. --help and -h are never written in the source — argparse adds them. If you extract flags naively and then verify against that list, a correct README that documents --help gets rejected as hallucinated. I had to add them explicitly as known-implicit. The environment variables were worse. The code reads them in a loop: for key in "PED_EDITOR", "VISUAL", "EDITOR": rv = os.environ.get(key) A regex looking for os.environ.get("SOMETHING") finds nothing here. My first verifier flagged PED_EDITOR as invented — in a README that was correct. The extractor was wrong, not the text. That failure was useful. It's the exact shape of error that makes verification worth building: the checker caught a mismatch, and the mismatch turned out to be in the checker. What it produced The full output is here. A few things it got right that I didn't expect: The environment variable precedence. It documented that PED_EDITOR takes priority over VISUAL, which takes priority over EDITOR — as a table. That ordering is only visible from the loop above; nothing states it in prose. PED_OPEN_DIRECTORIES. An undocumented-in-README variable that changes whether a package opens as a directory or as its __init__.py. It's one line in the source. The agent found it, explained what it does, and gave a shell snippet for setting it. Tab completion. It read install_completion.py, worked out that the module prints a completion script to stdout and picks bash vs zsh from $SHELL, and wrote a section for it. How it works. Four steps, each traceable to actual code — sys.path manipulation for pipx installs, difflib for partial name matching, inspect for locating source and line numbers, and the specific list of editors that get a +lineno argument. What went wrong One thing, and it's instructive. The Bash tab-completion section came out as: python -m ped.install_completion > /usr/local/etc/bash_completion.d That redirects into a directory, which fails. My first instinct was that the agent had invented it. It hadn't. That exact command is in the upstream README. The agent reproduced the source faithfully — including its mistake. This is a real limit of the approach, and I'd rather state it than hide it: the pipeline verifies that claims match the code, not that the code's own documentation is correct. When a project's README has an error, a code-grounded rewrite can carry it forward. I changed it to a generic form that can't be pasted into an invalid path: # Redirect the output to a file inside your shell's completion directory. # The exact location depends on your OS. python -m ped.install_completion > /ped I didn't invent /ped as the filename, because nothing in the code specifies a destination — install_completion.py just writes to stdout. Where it goes is the user's choice. The verification output cli_flags mentioned=8 invented=[] env_vars mentioned=5 invented=[] sections 12 trailing newline: ok requires_python stated: yes Zero invented flags, zero invented environment variables. Not because the model is trustworthy — because anything it invented would have been caught and the whole deliverable rejected. Cost $0.04 for the generation. One call, ~3,500 output tokens. The reading and verification added no LLM cost; they run as deterministic code. That ratio matters: the expensive part is the writing, and the writing is the part you can't trust without checking. What I'd take away from this Extraction is harder than generation. The model wrote a good README on the first try. Getting a reliable list of what's actually in the code took several iterations, and in this run the bugs I found were in my parser rather than in the generated prose. Verification changes what you can ship. Without stage 4, this is a demo. A README that's 95% right is unusable for anything real, because you don't know which 5%. With a checker that rejects unverifiable claims, the failure mode becomes "output rejected" instead of "output subtly wrong." Faithful isn't the same as correct. The completion-path bug is the clearest lesson here. Grounding output in source code means inheriting the source's errors too. A human still has to read the result. I turned this into a small service: send a public GitHub repository URL, get a README back within 24 hours, £30, and you see the full file before you pay anything. It's run by an AI agent that I supervise — I read every deliverable before it's sent. Details at agent.aitomaton.dev. The example above is the unedited pipeline output, apart from the one correction described. Compare it with the repository's own README if you want to judge for yourself.