A reviewer's attention is a fixed budget

The PR size grew significantly this year. But a reviewer can still only pay attention to a finite number of changes. For the past months we have been interviewing the tech leads who review AI-generated code for a living, and tinkering with our own review surface around what they told us. These are our conclusions so far. PRs are now larger. Human attention hasn't grown Every quote below is verbatim from the people we interviewed for our last post: four months, more than 100 tech leads, architects, and senior engineers, on how their teams actually use AI. "Now that you remove the bottleneck of writing the code, we've just moved the bottleneck to reviews." — an engineering manager The surge of coding agents has created strong control concerns for tech leads. Many try to limit this loss of control by reviewing as many PRs as possible. But that is where the bottleneck appears. What makes this a losing battle is that it isn't only a question of time. It's a question of cognitive capacity. No matter the skill level, no matter how much of a superhero you are, nobody is built to build a global understanding of a 2000-line AI-generated PR touching 100 files. The human attention window is limited. Three problems with the GitHub PR review interface for AI-generated PRs Every file has the same weight and is presented in simple alphabetical order. It shows what changed and not why. As reviewer, you must reconstruct the reasoning from the code, one file at a time. Not everything is in GitHub. You must keep GitHub open to comment, then have a local Claude Code session in another window, then your IDE again in another window. You constantly switch from one to the other. In the rest of this post, we'll describe what we built to address these 3 problems that came up again and again in our interviews. We're being fully transparent about what worked, what didn't, and what we still want to test. And of course, we're very open to feedback. 1. Triage by criticality In the last post we said that industry-standard code, a boilerplate integration or reading a CSV in Python, is easy for a model, and that as models commoditize, that kind of code commoditizes with them. This is the first point towards deciding what we should review and what we should not. The seniors we sat with already split their reviews in two, although they do not do it consciously. Risky code gets read first, every time. Depending on the profile of the reviewer, their definition of risky code changes. For a backend, the data model, the APIs, the contracts between services, auth, payments, anything that touches money or identity. This risky code is read first, every time. One CTO wrote their order down for us in a Slack message. "1. Check we haven't dumped [junk] into the data models. 2. Check we haven't modified contracts between micro-services 3. Then, once those parts are validated, I look at the coherence and readability of the code." Boilerplate is skimmed over. This type of code gets the last part of the review, when the reviewer is already not really paying attention anymore, e.g., the helpers, the tests, the connectors to a well-documented API. If it is not business critical, it is not your expertise, and it does not need to be non-standard; the model has seen the pattern ten thousand times and it will probably produce code that is good enough. What we built The first thing we built is a change map. Every changed file gets a criticality tier, and the tiers are set per viewer rather than per repo: a backend reviewer marks the data model and the APIs critical and the frontend tests low; a frontend reviewer flips the same PR the other way. This is done by matching file paths with rules. Before reading a line, you see the shape of what you are about to spend attention on, and you discard what you will not open. the change map on a 93-file PR — 6 critical / 22 high / 5 medium / 50 low / 10 needs triage] "Where I'm losing time is I need to skim through the changes to see what is touchy in the PR, now I can just see in a minute. This PR has critical files touched, or it's just low impact." — a lead architect Result: we shipped the criticality ordering and triage first: you can discard files you don't want to read, then the diff is displayed with critical files first, noise last. Users do like it, although they feel a bit uncomfortable explicitly saying "I'm not reading this" at first. Our users asked for the alphabetical file tree to navigate as well, because that is what they are used to, so we provide both and they choose. Either you navigate via alphabetical order, or you navigate by criticality. This is not perfect yet. A one-line change in a utils file can be the whole blast radius, and a path rule will not flag it. Our belief is that those changes are being treated as the last part deserving attention and thus not read carefully anyway. The tiers do not make it worse. A way to improve would be to make the rules smarter and flag tiers based on the semantic meaning of the changes. Another addition we're looking at is a clean way to mark "this we did not really check, and that is fine according to our policy." We have not shipped that yet. 2. Show the why behind the changes The reasoning that used to live in a dev's head, or on a whiteboard when the feature needed some more design, the edge cases, the alternatives that were rejected, the constraints, now gets typed into a prompt before the agent writes a line. Recovering what the human thought about and determining if the logic is correct has always been the slow part of the review. Now the issue is that there are many more decisions in a single PR, and you do not know whether a change was backed by actual thinking from the developer or was an arbitrary choice by the agent. "I spend my time on PRs writing why, why, why [...] I'm forcing the devs to own their decisions." — a tech lead What we built Hervé captures the developer's thinking (the prompts) and puts it next to the diff in each PR. That way, we can expose important decisions that were made during implementation, e.g., we cache X but not Y. Our first version surfaced all of it at the top of the page next to the PR description. That was too much, and a user told us so within a week: "The info at the top is just too much and I just skip it." — a CTO It felt like reading the PR twice. The second version keeps only what deserves the top of the page: the intent shifts, where the agent added or dropped scope against the spec it was given, and the undiscussed decisions, where the agent chose something on its own without asking the developer. Everything else is available on demand. Hervé can read the AI sessions behind the PR, so you can ask it whether the developer prompted for a given edge case, and it finds the answer in the session, or tells you it is not there. an undiscussed decision card — the agent added a cache on its own, and no turn in the session asked for it One thing that surprised us: the PR author finds it useful to read the undiscussed list, to see what the agent decided on their behalf. 3. The IDE feeling in the review surface Reviewing one PR today takes three tools. GitHub or GitLab, to read the diff and leave comments. An IDE on a local checkout, to navigate the code: go to definition, find usages, search for a symbol. And a local agent, to ask the higher-level questions about the change. Three applications for a single review. "I checkout locally on the PR to get the code in the IDE, asks questions via Claude-code, and make comments on Github. The worst is when I review multiple PRs in parallel, I start mixing the windows and do not know which is which anymore." — a lead dev at a startup What we built We built the IDE features the reviewer needs into the review surface: go to definition, find usages, symbol search, on the review page. That folds two of the three tools into Hervé. The third, the local agent/harness, we have not folded in yet. It comes with a subscription most developers already pay for, so one more question about a PR costs them nothing, and we have not found a way to do better or cheaper than that yet. For now, we have decided not to compete with it. Developers love their harness anyway. In fact, they're asking us to expose Hervé's features to their harness rather than trying to replace it for review. This is probably where we will be going. Conclusion We are asking people to digest an amount of information per PR that is far more than a head was built to hold. The tooling has not caught up to the AI revolution. We have been coming at it from a few angles: criticality and ordering, so the attention goes where it counts; the captured intent, so less of it goes to reconstructing the why; the IDE features, so it stops requiring three windows for a single review. Some of it worked, some of it did not. What caught your attention in our approach? Would you rather have a new review interface, or more information available through the CLI so your local Claude sessions can access the insights and the captured intent? You can try Hervé!