Reviewing AI-written code — Selena Lab

What we were testing

Reviewing AI-written code is the part of the workflow that decides whether the rest of it is worth anything. This note records one test of a two-agent review setup, including the part that failed: two agents produced confident, plausible and wrong explanations of the same problem before anyone opened the file that settled it.

Experiment notes in this Lab are published only when the setup, the inputs and the observed outcome can be reproduced. This one can. It is a single observation rather than a study: n = 1.

The hypothesis was that if two independent agents read the same source instead of reading each other's summaries, review quality improves and the owner stops acting as a message bus between chats.

Before and after. Reports used to travel between tools by hand; now the task, the code and the machine checks live in GitHub and each tool reads them itself.

Setup

Claude Code runs from a workflow file using the published GitHub Action, and is triggered by writing @claude in a comment on a pull request or an issue. It is not fully automatic, but from that point on it reads the pull request, the diff and the CI results itself rather than receiving a pasted summary.

The repository is a fork. It inherited the upstream project's workflows, including a Contributor License Agreement check.

The review happens twice: first the plan, before a line of code exists, then the code itself — with both reviewers looking at the same commit.

Component	Role	Auth
Project workspace	Holds documents, data and the specification	Subscription
Codex	Writes code and migrations; reviews its own diff	Subscription
Claude Code	Independent audit and security review, running as a GitHub Action	Subscription OAuth token, no API key
GitHub	Shared source: task, code, diff, machine checks	—

The four components and what each one was responsible for.

Inputs

The whole experiment ran against four artifacts, each of which is still in place and can be inspected again.

One pull request, open since 26 August 2026.
One consistently failing status check: Verify CLA signature.
The repository's commit history.
The CLA workflow file and the contributors file it reads.

Two confident explanations, both wrong

The first agent reported that the red check was cosmetic, that it always fails for agent-authored commits, and that it could be merged past. Plausible: a previous pull request had indeed been merged with the same red mark.

Asked to verify that, the second agent examined the commit history, observed that owner commits and agent commits carried different author addresses, and concluded that the check compares the commit author against the CLA signatory — fixable with one line of git configuration. Also plausible. Also wrong.

Neither agent had opened the workflow file. Both were reasoning about what a check of that name would probably do.

What the file actually said

The check does not read commit authors at all. It takes the GitHub login of the pull request author and looks for it in the contributors file.

That file is the signature registry of the upstream project the repository was forked from. It lists ten logins belonging to that project's contributors. The current repository owner's login is not among them, and adding it would mean signing another company's legal agreement.

So the check will stay red permanently. It is not fixed by a signature. It is fixed by correcting the inherited workflow — whose own header comment states that repository owners and collaborators are exempt, while its script exempts only bot accounts.

AUTHOR: ${{ github.event.pull_request.user.login }}

The line that settled it, from the CLA workflow.

Result

The hypothesis held in a narrow sense and failed in a broader one.

It held in that the process which produced the correct answer was reading the source. It failed as a claim about agents: adding a second agent did not produce the correct answer. Two agents produced two confident wrong answers, and the correct one came from opening the file.

The practical rule we took from this is not "use two agents". It is that an explanation is not evidence, however fluent it sounds, and that both reviewers must be pointed at the same artifact — the same commit, the same file — or their agreement means nothing.

Limits

This is one incident, and it should be read as one.

n = 1. It shows that this failure mode exists, not how often it occurs.
Not a controlled comparison. The two agents received different prompts and had different access. Nothing here compares model quality.
Not evidence that agents are unreliable in general. It is evidence that reasoning about a file without reading it is unreliable — which is equally true of people.
The always-red check is a contributing cause. A status that is permanently red stops being read. That failure is organisational, not technical.

What remains unknown

Two questions this experiment raises and does not answer.

Whether a second agent adds accuracy when both are explicitly required to cite the source lines they relied on. Not tested here.
How often plausible-but-unsourced explanations survive a two-agent review when nobody opens the underlying file.

How to reproduce

Any repository forked from a project that carries its own CLA workflow will reproduce this.

Fork a repository that carries its own CLA workflow.
Open a pull request from an account not listed in the upstream contributors file.
Ask an agent why the check fails, without giving it the workflow file.
Ask a second agent to verify the first, again without the file.
Read the workflow directory yourself and compare all three answers.

Reviewing AI-written code: what two agents got wrong