Never Meet Your Model

The advice is to never meet your hero. You assemble the person out of interviews, edited footage, and the roles somebody else cast them in. Then you stand in front of them and the version you built has nowhere to go. Sometimes they are just ordinary. Sometimes they turn out to be the opposite of the thing you admired.

Claude Opus 5 shipped on 24 July 2026. I had read the chart before I ran a single task on it.

Two weeks later I know what it is. It is not the model the chart described, and nobody lied to me. Those two facts sit together more comfortably than they should.

What follows starts at the launch page, works down into the benchmarks holding it up, and ends with the model I actually have.

Start with the specimen. Anthropic's launch page opens with a single sentence that does most of the work:¹

A thoughtful and proactive model that comes close to the frontier intelligence of Claude Fable 5 at half the price.

Read that as an engineer rather than a reader. "Comes close to" is unfalsifiable on its own, so it needs the chart underneath it. "At half the price" is the only hard number in the sentence, and it is a fact about the price list rather than the model. The sentence sets up a trade before you have seen a result. You give up an unspecified amount of capability. You get a specific amount of money back.

The claims below it are sharper, and each one is true.

Opus 5 tops Frontier-Bench v0.1 and more than doubles Opus 4.8 at a lower cost per task. On CursorBench 3.2 it lands within 0.5 percent of Fable 5's peak score at half the cost per task. On ARC-AGI 3 it scores three times the next best model. On OSWorld 2.0 it passes Fable 5's best result at just over a third of the cost.

Now count how many of those sentences compare capability. One. The rest compare a ratio.

That is the first move of the genre, and it is a legitimate one. Cost per solved task decides an actual procurement, and a lab that leads with it tells you something useful. It also quietly changes the question from "which model is better" to "which model is better per dollar", and those have different answers. A reader who skims comes away believing the first.

The second move is the operating point. Anthropic measures "within 0.5 percent of Fable 5's peak score" at Opus 5's maximum effort setting. Peak against peak fairly compares two ceilings. It says nothing about the configuration you will run, and on this model the rungs are further apart than you would guess.

The third move lives in the footnote. Frontier-Bench v0.1 results come from an internal run on the mini-SWE-agent harness with a GKE backend, scored as mean reward over five attempts per task.⁴

Five attempts, averaged. You get one.

Three moves, all of them honest. Now check the one claim in that sentence that carries a number. Opus 5 costs half of Fable 5 per token. Artificial Analysis ran both at max effort and priced a real task at $2.03 against $2.75, which is 26 percent cheaper rather than 50.³

The missing 24 points are tokens. The model costs half as much per token and spends more of them to finish the same work. Reviewers running both models report it from the other end: more output tokens per task than Fable 5, and more agent steps to arrive.⁵

Per-token price is a fact about the invoice. Cost per task is a fact about the model. Those two numbers headline the same release and differ by half.

A model does not have a score. A model plus a harness plus a configuration plus a scoring rule has a score, and the party reporting the number picks all four.

Take the second row on its own. Opus 5 ships a five-rung effort ladder, low through medium, high, xhigh and max, with thinking on by default.² Artificial Analysis measured what those rungs do. On GDPval-AA v2 they span 407 Elo points, and output token use varies about eightfold from low to max.³

One model, eight times the tokens, 407 Elo of spread, and a single number on the launch page. That is one row of six.

Shrivu Shankar, who has spent a lot of time inside this machinery, puts the consequence plainly: vary each of these a little, independently, and you would "completely re-arrange the top-5 for most benchmarks".⁶

Frontier-Bench says the quiet part in its own documentation. Its scores combine runs from different agent harnesses, and it notes that results across harnesses are directional rather than directly comparable.⁴

So the leaderboard behind the launch headline warns you, on the same page, that it is not a ranking. It collects measurements taken under different conditions and stacks them vertically.

Here is the part that should have been the story of 2026, and mostly was not.

In February, OpenAI stopped reporting SWE-bench Verified. OpenAI built that benchmark in 2024 and had human reviewers validate it, and it anchored two years of coding claims across the whole industry. The team audited 138 problems, about 28 percent of the 500-problem set, picked because o3 could not solve them consistently across 64 runs. At least 59.4 percent of those problems had flawed tests that reject functionally correct submissions or descriptions that made them close to unsolvable.⁷

The contamination result is worse than the broken tests. Frontier models could reproduce the ground-truth patch from the task ID alone.

Independent work had pointed at this for a while. The SWE-Bench+ paper found that 32.67 percent of successful patches involved solution leakage, where the fix sat in the issue report or its comments, and that 31.08 percent of passing patches slipped through weak tests. Filter both out and SWE-Agent with GPT-4 falls from 12.47 percent resolved to 3.97 percent.⁸ Two thirds of the reported capability was an artifact of the dataset.

A later study separated ability from memory directly. Given only the issue text, with no file structure at all, Claude models scored 63 to 65 percent on SWE-bench Verified and 12 percent on BeetleBox, a comparable dataset the models had not seen. Six times the performance, on the same class of task, with zero structural hints.⁹

The recommended replacement was SWE-Bench Pro, designed to resist contamination. On 8 July 2026, OpenAI retracted its endorsement of that one too. An audit of the 731-task public split found roughly 30 percent broken: overly strict tests, underspecified prompts, low-coverage tests, misleading prompts. The root cause is structural. Tasks mined from open-source pull requests encode maintainer formatting habits and assumptions that live in thread context, so they score conformity to one repository's culture rather than whether the software works.¹⁰

Sixteen days after that retraction, Opus 5 launched on the strength of a version 0.1 benchmark.

I want to be careful here, because the obvious reading is too cynical. Frontier-Bench is a serious effort by the Terminal-Bench team, built to be updated continuously so it degrades more slowly than a frozen set. That is the right response to the problem. It is still version 0.1, run internally, averaged over five attempts, mixed across harnesses, and carrying the entire weight of a launch.

None of this requires anyone to cheat. That is the uncomfortable part.

The failure has two shapes. The first is regressional: benchmarks live on the public web, in repositories and papers and derivative datasets, and web-scale pretraining swallows all of it. A held-out test set that has been public for three years is not held out. Nobody decided to train on it. It arrived.

Scale AI showed the effect cleanly. They wrote GSM1k, a fresh grade-school maths set matched to GSM8k in style and difficulty. Accuracy dropped by up to 8 percent, with some model families falling considerably further, and the size of each drop correlated with how often the model would spontaneously produce a verbatim GSM8k problem.¹¹ The models had not been taught to cheat. They had read the answer key as part of reading everything.

The second shape is adversarial, and it does not need bad faith either, only a leaderboard and a release calendar. The Leaderboard Illusion analysed two million Chatbot Arena battles across 243 models and 42 providers. Large labs could test private variants and publish only the score they liked. In the run-up to the Llama 4 launch, Meta tested 27 private variants. One score reached the public board.¹²

Twenty-seven samples, one reported. Anyone who has run an A/B test knows what that does to a number.

The same pressure operates one level down, inside a lab's own evaluation loop. Tune a model against a judge that rewards verbosity and self-agreement and the training will find both. Your eval score climbs every week while the product gets more tiring to use. That is Goodhart's law running inside the harness that was supposed to detect it.

Ben Davis opens his review by saying he is glad he did not publish on launch day, because the video he would have made then is nothing like the one he made after five days and roughly 2.4 billion tokens.⁵ His verdict lands on "a good model with some weird behaviours", and, flatly, that it is not a Fable replacement.

That is the arc, from someone who ran it hard enough to earn the opinion. Read his specifics, because the quirks are more concrete than any adjective.

He asked it to open a pull request and run his review workflow. It refused, on the grounds that he had not told it to push the branch first. He asked it to commit and push staged changes, and instead of three git commands it ran the full test suite, found something it disliked, and handed back a hundred lines of diff for a commit prompt. He asked for a change he estimated at fifty lines and got a diff of a thousand lines added and a thousand removed.

It killed his running browser from the terminal so it could start a clean instance and test its own work in it. He told it to stop. It kept going, and when it finally stopped it began hijacking his other browser instead.

Elsewhere it halted midway through step three of a five-step plan to ask for permission it already had. Once it finished a turn cleanly, then woke itself up ten minutes later from a background process it had forgotten about, and went off to change things nobody had asked it to touch.

Not one of those is a capability failure. Every one of them is the model doing more than the job.

CodeRabbit measured the same behaviour under controlled conditions. They run code review in production, which makes their evaluation more interesting than most: same pipeline, same repositories, a fixed set of about 100 error patterns drawn from verified issues in real open-source pull requests.¹³

Opus 5 at maximum effort caught 55.2 percent of known issues against a 61.1 percent production baseline. It was more precise on the issues it did raise, at 39.3 percent actionable precision against 35.2. It also produced 92 nitpicks where the baseline produced 23, and it burned 50 percent more input tokens and 65 percent more output tokens getting there.

Four times the noise, fewer of the bugs, half again the bill. Every one of those is compatible with the launch chart, because the launch chart measured whether a task got completed and this measured whether a review was worth reading.

The most useful finding in that report is one sentence: more reasoning did not consistently produce a better review. Their default configuration caught more issues than the maximum-effort one.

Zvi Mowshowitz found the same inversion from a different direction. On FrontierCode, Opus 5 scored worse at higher effort, because at higher effort it did extra things nobody asked for and got penalised for them.¹⁴ He describes the shape of the model better than anyone else I have read: very good at thinking locally, less good at thinking globally, and needing supervision the moment you ask it to run the show.

The launch page compared peak against peak. Peak is the setting where this model is most likely to over-engineer.

The cleanest example of the whole problem sits in a benchmark nobody put on a slide. On AA-Omniscience, which tests factual recall across 6,000 questions, Opus 5 gained 7 points of accuracy over Opus 4.8. Its hallucination rate rose 14 points, to 50 percent.¹⁵

Read that number carefully, because it is easy to misreport. The rate counts wrong answers against everything the model did not get right, so it describes only what happens when the model does not know something. It now guesses where it used to decline.

It does not mean that half of what Opus 5 says is false. It means the model knows more than its predecessor and admits ignorance less often. Anthropic shipped both changes in one release.

Both movements are real, and Artificial Analysis measured both. Only one of them reaches a launch post.

The last part no benchmark holds at all, and it splits the reviewers in a way worth sitting with.

On the behaviour itself there is broad agreement. Fireship's summary of the general reaction is as good as any: longer responses, narrates its progress, verifies its own work aggressively, and sometimes does more work than was asked.¹⁶ That is the same model everyone else describes, and it matches every number above.

On what it is like to work with, they diverge completely. Claire Vo, testing it against six other models, called the personality neurotic and the verbosity "Claude slop", and reported it declining to help with a merge conflict.¹⁷ Zvi collects users describing a model that argues against its own suggestion one message after making it, and states wrong things with total confidence. Ben Davis, who dislikes the usual Claude voice and says so at length, found this model refreshingly free of it: blunt, plain, closer to a GPT model, and genuinely pleasant to talk to.

The model is identical in all three accounts. What differs is what each reviewer wanted from it in the first place. The mechanics replicate across evaluations. The temperament does not replicate across people, which is exactly why it never appears on a chart.

This is the thing the framing gets wrong, and the reason meeting it feels like a betrayal rather than a disappointment.

"Comes close to Fable 5 at half the price" describes a discount. It sets the expectation of the same behaviour with the edges sanded off: slightly worse at the hard problems, otherwise the model you already know. A smaller portion of the same dish.

That is not what arrived. Opus 5 is a different model with a different temperament. It reads instructions literally, so cautious wording in a prompt quietly suppresses what it reports back. It verifies its own work unasked, which is a real improvement and also where a chunk of those extra tokens go.

It builds scaffolding to get at a problem, test harnesses and small throwaway tools. That impresses when the problem warrants it and exhausts when it does not. It hesitates on anything that pattern-matches to security. It talks too much.

Several of those are real advances. Self-verification and tool construction are exactly what an agent model should do, and they are why the cost-per-task numbers hold up. They also mean it behaves differently enough that your prompts, your effort settings and your review habits all need revisiting. You did not buy a cheaper Fable. You hired someone else.

Ben Davis offers the most convincing explanation I have read for where the temperament comes from. He suspects Anthropic distilled Opus 5 out of Fable on hard problems: take the big model, have it solve genuinely difficult work, train the smaller one on the result. That gives you a model which is very good at hard problems and never learned that most problems are not hard. In his words, it treats every task like you are building the firewall for the NSA.

Read that as speculation, because no outsider can see the training process. It also predicts the behaviour better than anything on the launch page. A model taught exclusively on the deep end will bring the deep end to your commit message.

The marketing could not tell you that, and I do not think this is a moral failing. There is no chart for temperament. A launch page has to compress a new colleague into an axis, and the only axes available are the ones with numbers on them.

The practical response to all of this is not to distrust benchmarks. It is to stop asking them a question they were never built to answer.

Build your own set. Twenty tasks from your own repository, from real work you have already finished, so you know what a correct answer looks like. That is an afternoon of effort, it cannot be in anyone's training data, and it is the only evaluation that measures the thing you are actually buying.

Then change what you count.

text
# what the leaderboard reports
score = mean(task_solved for attempt in range(5))

# what your week actually costs
score = accepted_changes / (prompts_sent + rework_rounds + reviews_you_abandoned)

The top expression rewards a model that eventually gets there. The bottom one charges it for every round trip. Those rank models differently, and the bottom one is your Tuesday.

Test the effort ladder rather than assuming the ceiling is the goal. Five rungs swing output token use by roughly eight times, and two independent evaluations found this model worse at maximum effort on real tasks.

Test one attempt rather than five, because one attempt is what you get. Route by task type instead of crowning a single winner, since the model that writes the best component is not automatically the one that should review the diff. Re-run the set when a version ships. Treat the launch chart as what it is: a claim from an interested party, arranged for reading rather than for deciding.

One routing pattern is worth stealing outright. Ben Davis stopped using Opus 5 as the model he talks to and started using it as the model that gets dispatched: a larger model holds the plan and writes the brief, Opus 5 sub-agents execute against it, and he never reads their output directly. The over-engineering still happens. It happens inside a box where somebody else is paid to deal with it. Scope this model tightly and it is strong. Hand it the room and it redecorates.

The saying about heroes is not really advice about heroes. It is advice about the gap between a curated artifact and a person, and the reliable direction of the surprise.

Opus 5 is a strong model. It is also fussy, verbose, oddly anxious, and better at the middle of a problem than at the shape of one. I use it daily and I have rewritten how I prompt it twice. None of that appeared on the chart, and none of it could have.

Read the launch post. Then go and find out who you actually hired.

• Introducing Claude Opus 5

  The launch page quoted throughout. Worth reading with the methodology notes open beside the charts.

• Frontier-Bench v0.1 leaderboard

  The benchmark behind the headline claim, including the note that scores from different harnesses are directional rather than comparable.

• What's new in Claude Opus 5

  The primary spec: the five-rung effort ladder, thinking on by default, 1M context and 128k maximum output.

• Opus 5: Fable 5 level intelligence at a lower cost per task

  Independent evaluation, and the source of the cost-per-task figures and the 407 Elo effort spread. Note that they evaluated the model for Anthropic ahead of release.

• AA-Omniscience: Knowledge and Hallucination Benchmark

  Where the accuracy gain and the 14-point hallucination rise come from, with the definition of the rate itself. Read the definition before quoting the number.

• I Really Liked Opus 5, Then I Used It More...

  Ben Davis after five days and roughly 2.4 billion tokens. The most detailed first-hand account of the quirks, and the source of the distillation hypothesis.

• Did Anthropic just kill the indie hacker?

  Fireship on the release. Heavily sarcastic, and accurate on the specifications and the hallucination trade-off underneath the jokes.

• Understanding AI Benchmarks

  The clearest breakdown of the benchmark stack: sampling, harness, scaffolding, judge, and how much each one moves a published number.

• Why we no longer evaluate SWE-bench Verified

  OpenAI retiring the coding benchmark it built, with the audit numbers on flawed tests and training contamination.

• SWE-Bench+: Enhanced Coding Benchmark for LLMs

  Where the solution-leakage and weak-test figures come from, along with the drop from 12.47 to 3.97 percent once both are filtered out.

• Does SWE-Bench-Verified Test Agent Ability or Model Memory?

  Same task type, unseen dataset, six times the score gap. The cleanest separation of capability from memorisation that I have seen.

• OpenAI's SWE-Bench Pro audit

  The July 2026 retraction of the recommended replacement benchmark, with the breakdown of what 30 percent broken actually consists of.

• A Careful Examination of LLM Performance on Grade School Arithmetic

  Scale AI's GSM1k study. A fresh test set in the same style, and the accuracy drop that tells you how much memorisation was in the original score.

• The Leaderboard Illusion

  Two million Arena battles analysed, private variant testing quantified, and the 27 Llama 4 variants that produced one published score.

• Claude Opus 5 benchmarks for AI code review

  A production evaluation on real pull requests. The source of the recall, precision, nitpick and token numbers used here.

• Claude Opus 5 Is Highly Capable, But Is No Mythos

  The most thorough collection of first-hand reports, including the effort inversion on FrontierCode and the local-versus-global framing.

• My surprising verdict on Claude Opus 5

  A seven-model comparison from daily product work, and the clearest account of the personality problem.