2026
2026년 9월 18일
“This AI model is good.” By whose standards?
Why the same AI model can be useful to one person and unacceptable to another. Research, user reports, and a practical way to test your own quality threshold.
소프트웨어 개발자
약 11분
2026년 9월 18일
Why the same AI model can be useful to one person and unacceptable to another. Research, user reports, and a practical way to test your own quality threshold.
소프트웨어 개발자
약 11분
Two developers look at the same AI-generated export script. It produces the requested file. One calls the model excellent. The other would never let that code into their project.
In this hypothetical case, the first needs a one-off export from known data and checks the result by hand. The second needs a reusable feature that preserves permissions and handles missing fields. The same script satisfies one brief and fails the other.
“This model is good” becomes useful when you add for which work, under whose acceptance rules, and with how much help. Models differ in capability. Our requirements determine which differences matter. Sometimes one person's bar is higher. Sometimes they care about another dimension. Sometimes someone has missed a defect.
The useful comparison starts by making those conditions visible.
Your baseline is the alternative already available: your own work, a colleague's help, a search engine, or leaving the task undone. Your bar is the minimum result you will accept for this task.
Someone writing an email in a second language may gain a lot from a plain, accurate draft. A professional editor may reject the same draft for flattening the writer's voice. The improvement is real, and so is the work that remains.
Brynjolfsson, Li, and Raymond studied an AI assistant's staggered introduction among 5,172 customer support agents. Issues resolved per hour rose by 15% on average. Less experienced and lower-skilled workers gained more. The most experienced and highest-skilled workers saw small speed gains and small quality declines.1 In that support setting, the average concealed differences across skill and experience.
An improvement over your baseline can still fall short of the job. A beginner's much-improved translation may need an expert before publication. Conversely, a model can meet every requirement and save little time because you already finish that work quickly. “Better than my alternative” and “ready to use” need separate answers.
Some requirements are plainly stricter than others. Handling ten known inputs demands less than handling arbitrary customer input. A rough draft allows more unfinished work than a message ready to send.
Other differences have no useful up-or-down ordering. A novelist may demand distinctive dialogue while tolerating slow responses. Someone drafting routine replies may value speed and consistency more. Working offline imposes a constraint that a benchmark score cannot remove.
| Task | Accept | Reject |
|---|---|---|
| Invitation ideas | A usable starting point | Wrong tone |
| Email rewrite | Facts and meaning preserved | Invented promises |
| One-off export | Correct on the actual data | Missing or altered records |
| Shared application | Behavior and project rules hold | Broken permissions |
These are illustrative rules. The person writing the invitation can also be responsible for the application. Their bar changes with the task.
Expertise cuts both ways. An experienced developer may recognize a subtle failure and reject the code. They may accept an imperfect draft because they can repair it cheaply. Someone unable to check that code might reasonably require more independent verification.
Accepting less polish is a choice. Overlooking an error means you have not established that the result meets your bar. Personal preference does not make a wrong calculation correct.
A benchmark fixes tasks and scoring rules to make comparison possible. Its winner is a useful candidate when those conditions resemble yours. A preference ranking answers another question: which outputs did its voters prefer?
A 2026 Findings of ACL study built personal rankings for 115 active Chatbot Arena users, each with at least 25 comparisons. Average correlation with the aggregate ranking was 0.04 using Bradley-Terry estimates and 0.43 using Elo. A value of 1 means identical ordering. A value of 0 means no rank association. The method-sensitive result supports variation within this selected sample, not stable rankings for everyone.2
Preference also cannot certify correctness. Smoother prose is no compensation for an invented promise in a customer email.
Public reports help identify what people actually value. On 17 October 2025, Reddit user nNaz described work on complex financial codebases. They used Sonnet 4.5 for research and implementation, but reserved Opus for turning the findings into a plan. They criticized project-wide understanding and restrictive usage limits.3
Five days later, ak47surve reported feeling as productive with Sonnet 4.5 as in their earlier Opus workflow. Avoiding frustrating limits was part of the benefit.4
Outside coding, Sammi_W preferred Gemini's experimental “1206” over “0205” for writing in a 7 February 2025 forum post. They cited repetition and weaker plot and character continuity in the latter.5 Their standard covered maintaining a story, something a single attractive paragraph cannot show.
These dated reports supply test criteria, not current buying advice. The self-selected accounts cannot establish prevalence or whose bar was higher. They lack a controlled comparison of tasks, settings, prompts, or checking, so they cannot isolate the cause of the disagreement.
Turn those criteria into questions about your work. Can the model plan across this project, preserve this character's voice, or remain available when needed?
A brilliant answer shows what a model can do once. A dependable workflow needs it to meet your requirements repeatedly, including on the unremarkable parts.
Return to the export script. Asking for a change to a known function leaves you responsible for finding the cause and checking the result. Delegating the whole problem adds investigation, planning, tests, and handoff. The individual steps need not become harder. The model becomes responsible for more places where something can go wrong.
Model A's higher peaks do not repair its failed requirements. Model B meets the bar throughout this illustration. In real work, the weakest result you have observed cannot guarantee a model's future minimum. Track how often it falls short, how seriously, and whether it catches and repairs its mistakes.
Time without intervention and quality at handoff are different measures. An agent can keep running without useful progress. One that stops and reports a missing permission leaves a visible, unfinished task. A silently incorrect result gives you another verification problem. More autonomy is valuable when it delivers acceptable work with less of your attention.
Imagine twelve stages, each succeeding independently with the same probability. Every stage must succeed, and there is no recovery. At 95% success per stage, the chance of a fully successful run is about 54%. At 99%, it is about 89%, calculated as 0.95^12 and 0.99^12.
These are invented probabilities. Real stages differ, failures can be related, and agents can recover. The calculation illustrates accumulated exposure to failure. It measures no model and predicts no autonomous run length.
A useful verdict concerns how often a complete assignment meets your requirements within your time and budget. Include failed attempts. One impressive demo, or one disastrous answer, cannot establish that frequency.
Your expectation is what you predict the model will deliver. It can differ from both your existing alternative and the result you need.
After repeated failures on broad assignments, you might restrict an assistant to small edits. If two newer models both handle those edits, the comparison tells you they suit that narrow workflow. It says little about the investigation or checking you stopped delegating.
Retest that boundary when the model or setup changes. Delegating fewer steps while keeping the same requirements narrows the job. Accepting worse results lowers the bar. Improving context or tools may improve performance against an unchanged bar. Mixing those decisions hides what actually changed.
Expectations can also make a weak result feel good. A 2026 preprint, revised in August, tested capability framing with 162 participants and six models. How researchers introduced a model affected impressions and interaction, without a detected effect of framing on output quality. The short tasks and US, English-speaking sample limit generalization.6 Check the work against the rules you set before the model had a chance to impress or disappoint you.
The cost of an acceptable result includes preparation, waiting, checking, corrections, and failed attempts. Token prices cover only part of it. A cheaper model may need more of your time. A more expensive one may add nothing when both already meet your requirements.
In Stack Overflow's 2025 survey, 66% of the 31,476 respondents to its frustrations question selected almost-correct AI solutions as a problem. That is a self-reported frustration, not an error rate for generated code.7 A defect you can fix in seconds has a different cost from one that takes an hour to find.
Your own time can be hard to estimate from memory. In METR's early-2025 experiment, 16 experienced developers completed 246 tasks in repositories they knew. Allowing AI increased completion time by 19%. Afterwards, participants estimated that AI had made them 20% faster.8 The finding concerns that setting and those early-2025 tools.
METR's February 2026 follow-up reported signs of greater benefit but called the new productivity estimate unreliable. Some developers declined to participate, and participants withheld tasks they did not want to do without AI. Lower pay introduced another possible selection effect, and concurrent agents complicated time measurement.9 Neither study supplies a universal speedup figure. Feeling less tired may itself be valuable, but it is a different outcome from finishing sooner.
Consider an illustrative choice: €1 and 25 minutes of your attention, or €4 and five minutes, with equal final quality. The extra €3 buys twenty minutes back. That may be worthwhile at work and unaffordable on a fixed cash budget. Saved time does not automatically become spendable money.
Compare the complete setup: model version, reasoning setting, tools, context, and your interventions. Repository access and working tests change the work an agent can complete. After a failure, inspect what it actually read, changed, and checked. Repair missing context or broken tools where the evidence supports that diagnosis. Some failures remain model limitations. Count the work of improving the setup, too.
A personal evaluation needs a few real tasks and acceptance rules written before you see the answers.
The result could be: “For routine replies in German, this model preserves the facts and needs only a quick wording edit. I still verify every date.”
Or: “For changes across this repository, the other model needs fewer corrections before the checks pass, enough to justify its cost.”
Each verdict gives the next person something to compare with their own requirements. Ask what work clears the bar, and how much human work it takes to get there.
A study of 5,172 support agents shows how the same assistant benefits workers differently.
Individual model rankings, aggregate rankings, and the limits of averaging preferences.
An experiment on how a model's introduction shapes the judgment users carry away.
Why selection effects make the follow-up difficult to interpret. Read alongside the 2025 result.