← Owen Carlsen

Fable 5.2 vs 5.1 on 124 bugs from my own pull requests

Owen Carlsen · Nov 2026

Anthropic released Claude Fable 5.2 on November 9. Fable reviews my pull requests (PRs), as a Claude Code subagent, so the first thing I wanted to know was whether to switch. Then I found that I had never pinned the reviewer's model. The reviews ran through Claude Code's fable alias. On the version I had, that alias resolved to Fable 5.1. The changelog for the release that adds 5.2 says the alias moves to 5.2. My reviewer would have changed on release day and nothing in my repo would have said so.

That is the exact failure I built pinbench to prevent. Pinbench is a command-line tool that compares two pinned models on my own tasks, and I wrote about pinning in Upgrading production models without guessing. So I pinned the alias to 5.1 before updating, with one setting: ANTHROPIC_DEFAULT_FABLE_MODEL=claude-fable-5-1. Then I measured the two models the way pinbench always does: through the Claude API, each model ID pinned, at the effort my reviews already ran at.

The first test was the wrong test

The quick answer came from pinbench's summary suite: ten short passages, a judge from another vendor, three key facts per passage.

Pinbench's verdict was RECOMMEND, meaning switch, but only for summaries. It says nothing about code reviews.

The review suite that ships with pinbench is six tiny diffs with one planted bug each, and tiny diffs never go to Fable in my workflow. Fable gets the PRs that touch several files and a few hundred lines, so I needed a test made up of those.

An answer key made of fixes

I ran a one-person company, now closed, whose operations ran on about a hundred scheduled jobs. Most of that code was written by AI agents, and PRs were reviewed by a model before they merged. A PR could be reviewed more than once, and each review was a round. When the reviewer found something, the agent that requested the review fixed it in a commit of its own, with a message like "review round 1: ..." naming what it fixed.

I used those commits as the answer key. For each one, I rebuilt the diff the reviewer saw just before the fix. A model then wrote down each defect the fix corrected, as a description of what was wrong in the code. On the PRs I tested, that came to 209 total defects, which I bucketed into blocking and non-blocking.

A blocking bug is one that should stop a merge: wrong behavior on a realistic input, lost data, a message sent that shouldn't have been, or a safety check that lets things through when it should block them. The rest are smaller defects like a stale comment or a missing test. The headline score uses the blocking bugs that are visible in what the reviewer was given, without running any code, of which there were 124.

Each model got what the reviewer had at the time: the diff, the full text of the changed files, and the ticket when there was one. It got nothing written after the review. Claude Opus 5.5 graded every review blind against the answer key, one review at a time. A review got credit for a defect only if it pointed to the same place in the code, and described the same wrong behavior, as the answer key. I used the same judge, with the same instructions, for every review.

What the live runs caught

I first ran a smaller, twelve-item version of this test, and it found four problems. I fixed each one before the full run:

1. Thinking used up the output budget. Both models think before they answer, and at the default effort (high), several reviews spent the whole 16,000-token ceiling thinking and returned nothing. Pinbench refused to score them instead of counting them as misses. The full run uses a higher ceiling and medium effort.
2. Each call took from 1-4 minutes, but pinbench had a fixed 60-second timeout (now configurable, issue #8). So, for the full run, I added Batch API support to pinbench. A batch has no per-call timeout, and it costs half as much.
3. The prompts I had first used gave away some of the answers. The review requests written at the time named what to look for, and more than half of the defects were hinted at this way. The final suite uses one neutral prompt.
4. Each model got the PR's diff plus the full text of some files. At first, the only files shown in full were the ones the fix changed: the buggy ones, in other words. That told the model where to look. Claude Opus 5.5 caught this in a design review. The final version shows the full text of every file the PR changed, the same files a reviewer would open, chosen without looking at the fix.

Results

Both models ran at medium effort (the API's effort setting), the level all my past reviews used. I ran each model twice, to see how much a score changes between two runs of the same model.

Running each model twice shows how much a score moves on its own: 5.2 found the same number of blocking bugs both times, and 5.1 found three more the second time.

With only 124 bugs, the gap between the two models has a wide margin of error. To size it, I used a bootstrap: 4,000 pretend versions of the test, built by drawing tickets at random from the real ones. In 95% of them, the range was +2 to +20 in the first run, and −3 to +19 in the second. So the second run's range includes zero: on that run alone, a tie can't be ruled out.

I read a sample of the reviews where the two models disagreed. 5.2's real results are, if anything, a little stronger than the numbers above: twice the judge marked a bug as missed that 5.2's review had in fact named, and the two judge replies that failed to parse were both on 5.2's reviews.

The table also shows 5.2 was cheaper per review. Here is where that comes from. Each review read the same input on both models, so the difference is in what they wrote back and what each token costs:

Input is a little over half of the bill, because every review reads about 46,000 tokens of diff and files, and the lower price applies to all of it. That is why 9% more output still came out cheaper. The whole run, both models twice plus the judge, cost about $158 at batch price.

The lower cost depends on the effort setting, though. At its default effort (high), on the earlier twelve-item run, 5.2 wrote about 70% more than 5.1. The break-even here is about 60% more, so at its default effort 5.2 would likely cost more per review. (I didn't measure whether that extra thinking finds more bugs.)

Caveats

• The answer key leans toward 5.1, because 5.1 did many of the original reviews. If it missed a bug, nobody fixed it, so the bug isn't in the key and 5.2 gets no credit for finding it.
• Each review was one prompt with no tools, so this measures reading a diff. In a real review, the model also reads the repository and runs tests.
• It is one codebase, mostly written by AI agents.
• I tested two days after release, at one effort setting.

It was enough for my own reviews, so I moved the pin with the same setting: ANTHROPIC_DEFAULT_FABLE_MODEL=claude-fable-5-2. I also kept CLAUDE_CODE_SUBAGENT_EFFORT=medium, so my reviews run at the effort I measured and not at 5.2's default (high).

Get in touch on LinkedIn

How this was written: Claude drafted from my notes; I edited the draft and checked every claim against the code and logs.