Your AI Model's 95% Accuracy Score Is Lying to You — This New Benchmark Proves It

You did everything right. You compared the leaderboards, picked the model with the highest accuracy score, and shipped it. Then it confidently gave a user the wrong answer at the worst moment. The problem: a wrong trivia answer and a wrong medical answer both count as "one error" on every benchmark you checked. In the real world, one is a shrug and the other is a lawsuit, a lost client, or a patient harmed.

A new paper calls this out directly. "UtilityBench: Measuring Language Model Utility When Errors Have Different Costs" by Qifeng Lei, Zhiyong Yang, Zitai Wang, Shilong Bao, Linchao Zhang, Qianqian Xu and Qingming Huang (arXiv:2610.25924, 19 October 2026) asks the question no leaderboard asks: not "how often is the model wrong?" but "how much does each wrong answer cost?"

What the researchers did

The team started from one observation: mainstream evaluation protocols treat all errors as equal, though the impact of a mistake "often extends far beyond what correctness can capture." In medicine, they note, a single error can put patient safety at risk. So they built a benchmark that prices errors instead of just counting them.

UtilityBench is 15,000 questions with verifiable answers, spanning mathematics, coding, natural sciences, law and medicine — domains where a wrong answer has real consequences. Instead of marking each response right or wrong, the researchers quantified how bad each kind of error is, using Best-Worst Scaling — a method borrowed from psychology in which people repeatedly pick the best and worst items from a set — to produce data-driven cost estimates. The result is a benchmark where every question carries a price tag for getting it wrong.

Those price tags decide when a model should answer and when it should abstain. Saying "I don't know" on a question where a wrong answer is expensive is the right decision — yet today's benchmarks never reward it.

To measure this, the paper introduces UtilityAUC, a metric that measures the area under the utility curve determined by a model's response ordering. In plain English: it checks whether the model puts its most reliable answers first and its shakiest ones where they do the least damage — or refuses them outright. The authors show theoretically that UtilityAUC can distinguish between models where standard accuracy and AURC (the area under the risk-coverage curve) fall short.

Evaluating 20 state-of-the-art models, the researchers found that greater accuracy or more advanced architectures did not reliably translate into higher UtilityAUC. The best problem-solver was often not the most responsible decision-maker — a gap between problem-solving ability and responsible decision-making that accuracy scores hide completely.

Why it matters

First, leaderboards optimise the wrong thing. Every public ranking used to pick a model is built on accuracy: right answers divided by total answers. UtilityBench shows that two models with the same accuracy can have very different real-world value, because one fails cheaply and the other fails expensively. If you choose by accuracy alone, you may be shipping the more dangerous model.

Second, abstention is a skill, not a failure. Most products treat "I don't know" as a bug. UtilityBench treats it as the correct answer whenever the cost of being wrong exceeds the value of being right. A support bot that refuses a question it might get wrong beats one that hallucinates a refund policy. A coding assistant that says "I'm not sure, here's what to verify" beats one that invents a subtly broken function.

Third, error cost is domain-specific, so your evaluation should be too. A wrong answer in a brainstorming tool costs almost nothing; the same error in a medical summary, a contract review or a tax calculation can be serious. A single global accuracy number cannot capture that. The right question is always: what does a mistake cost in this product, for this user?

How YOU can use it

UtilityBench is a research benchmark, not a library you can pip-install today. But its core idea — price your errors, then measure utility, not accuracy — applies to any product you ship this week. Here is how to use it.

Step 1 — Score your own evals by cost, not by count. Take the test set you already use for your product and add a cost column: 1 for a harmless miss, 5 for a confusing one, 25 for one that costs money or trust. Then compute a cost-weighted score instead of plain accuracy. The snippet below is illustrative, not paper code:

# Cost-weighted evaluation (illustrative — not paper code)
# cost: 1 = harmless miss, 5 = confusing, 25 = costly mistake
results = [
    {"correct": True,  "cost": 1},
    {"correct": False, "cost": 25},   # wrong medical dosage advice
    {"correct": False, "cost": 1},    # wrong trivia answer
]

total_cost = sum(r["cost"] for r in results if not r["correct"])
max_cost   = sum(r["cost"] for r in results)
utility_score = 1 - (total_cost / max_cost)

print(f"Accuracy: 33% — but utility score: {utility_score:.2f}")
# Two models can share 33% accuracy and still have
# wildly different utility scores. Pick the cheaper failure.

Run your two candidate models through the same weighted set. The winner is the one that fails cheap, not the one that fails least. When you compare models for a real deployment, pick by utility score: if the leaderboard winner loses on your weighted set, trust your set — it measures the errors your users will actually pay for. The paper's headline finding is exactly this — the most accurate model is not reliably the most useful one.

Step 2 — Give your model an abstention rule. Add this to your system prompt:

You are a careful assistant. Before answering, estimate your
confidence from 0 to 100.

- If confidence is 80 or above, answer directly.
- If confidence is 50–79, answer but start with
  "I'm fairly confident, but please verify:" and name
  exactly what to check.
- If confidence is below 50, say "I don't know — here's
  what I would need to answer this safely:" and ask for it.

Never guess a number, a date, a dosage, a clause or a
fact you cannot verify. A refusal is always better
than a confident wrong answer.

Then measure the refusal rate on your hardest questions. If the model never refuses, your threshold is decorative.

Step 3 — Rank models by utility, not accuracy. If the leaderboard winner loses on your weighted set, trust your set — it measures the errors your users will actually pay for.

Step 4 — Test refusal on purpose. Build a small "should-refuse" set: questions your product must not answer — medical advice beyond its scope, legal claims it cannot verify, questions missing context. A good model refuses most of them; a dangerous one answers them all confidently. Track this refusal rate next to accuracy in every model review. UtilityBench's lesson is that knowing when not to answer is part of intelligence, not a gap in it.

Key takeaways

• Accuracy treats every error as equal. Reality doesn't — a wrong medical answer and a wrong trivia answer are not the same mistake.
• UtilityBench (15,000 verifiable questions across maths, coding, science, law and medicine) prices errors using Best-Worst Scaling, so each mistake carries a data-driven cost.
• Its new metric, UtilityAUC, measures whether a model orders its answers by reliability — putting confident answers first and refusing or downgrading shaky ones — and separates models that accuracy and AURC cannot.
• Across 20 state-of-the-art models, higher accuracy did not reliably mean higher UtilityAUC. The best problem-solver is often not the most responsible decision-maker.
• Your move today: add a cost column to your evals, give your model an explicit abstention rule, and rank models by cost-weighted utility — not by the leaderboard.

Read the original paper: "UtilityBench: Measuring Language Model Utility When Errors Have Different Costs" by Qifeng Lei, Zhiyong Yang, Zitai Wang, Shilong Bao, Linchao Zhang, Qianqian Xu and Qingming Huang. Link: https://arxiv.org/abs/2610.25924 — the dataset is available on Hugging Face at https://huggingface.co/datasets/leiqf/UtilityBench