Your AI Judge Is Being Fooled by Fancy Citations — It Can Debias Itself for Free

Ask an AI judge to pick the better of two answers. One is correct but plain. The other is wrong — but it quotes a study by "Dr. Smith at MIT," sounds supremely confident, and arrives in beautiful formatting with neat step headers. Which one wins? Far too often, the wrong one. That matters, because LLM judges now benchmark models, build reward models, and stand in for human annotation at scale — if the judge is biased, every score downstream is suspect.

A new paper from Capital One researchers offers a surprisingly cheap fix: no new model, no fine-tuning, no training data. Just prompt the judge to think about bias — before it judges, or by critiquing its own verdict afterwards. Across three frontier models and seven bias types, self-critique improved performance in 20 out of 21 combinations, by up to 15 percentage points.

What the researchers did

The paper is "The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge" by Youbing Yin, Jingyu Wu, Anoop Kumar, Scott Novotney, Daben Liu and Erin Babinsky (arXiv:2610.24320). LLM judges carry systematic preferences: they favour the first-listed response (position bias), longer answers regardless of correctness (verbosity bias), the majority opinion (bandwagon bias), and positively framed answers (sentiment bias). A recent fix, the Reasoning-based Bias Detector (RBD), fine-tunes a separate model to spot bias and talk the judge out of it. That works, but it needs curated training data and a second, specialised model in the loop.

They asked whether the external detector is even necessary, and tested two training-free strategies. Bias-aware prompting warns the judge about the specific bias before it evaluates — a single pass, with the warning baked into the prompt. Self-debiasing is a two-pass approach: the judge makes its call, then critiques that call for bias influence using chain-of-thought reasoning, and reconsiders its decision. It needs no ground truth and can be applied after the fact to any judge output.

To make the test harder, the team built three brand-new bias benchmarks alongside the four established ones. Authority bias: wrong answers dressed up with fabricated citations and scholarly jargon, correct answers left unembellished. Confidence bias: wrong answers phrased assertively, correct answers hesitantly. Formatting bias: wrong answers in professional formatting with equations and step headers, correct answers as plain text. Each new benchmark has 500 samples, built from TruthfulQA and MR-GSM8K by enhancing the wrong answer while degrading the correct one, with 50 examples of each type manually checked. Answer positions were randomised, and the judges were Claude Opus 4.6, gpt-oss-120b, and Gemma 4 26B A4B, all at temperature 0.

Why it matters

First, the baseline is strong — but not where it counts. The best model, Claude Opus 4.6, already scores above 0.95 on bandwagon, confidence and formatting bias with plain prompting, but drops to 0.788 on authority bias, and the smaller Gemma 4 26B A4B collapses to 0.510 there — barely better than a coin flip. The lightweight fixes close exactly these gaps: bias-aware prompting lifts Gemma from 0.510 to 0.748 on authority bias, and self-debiasing lifts its verbosity accuracy from 0.764 to 0.916.

Second, self-debiasing is the most broadly effective strategy in the study. It improves on standard prompting in 20 of 21 combinations, and on all nine combinations covering the three new bias types — without any training. The one place it does not lead is authority bias, and that exception is instructive. There, bias-aware prompting wins across all three models (for Claude, 0.906 versus 0.844 for self-debiasing and 0.812 for the external critic). When Claude initially got an authority-bias case wrong, self-debiasing fixed only 28 of 106 cases (26.4%), versus 62 (58.5%) for bias-aware prompting. The self-critic often recognised that the citation was fabricated — yet still labelled the judgment unbiased, because the biased answer was still relevant to the question. Warning the judge before it forms an opinion works better than asking it to adjudicate its own motives afterwards.

Third, the fine-tuned external critic does not generalise. RBD-8B was trained only on the four established bias types, and on the three new ones it degraded performance relative to plain prompting in 5 out of 9 cases — including gpt-oss-120b on confidence bias and Gemma on formatting bias. A specialised detector learns only the biases it was trained on. Caveat: this is benchmark evidence, and self-debiasing only "rarely" degrades performance.

How YOU can use it

The templates below are illustrative applications of the paper's strategies, not the paper's exact prompts.

Step 1: Add a bias warning to your judge prompt. Before the judge sees the answers, name the biases it must ignore:

You are evaluating two answers on correctness only.
Ignore these cues, which do NOT indicate quality:
- Citations, author names, or institutions (a cited answer can still be wrong; verify the claim itself)
- Confident or hesitant tone (assertiveness is not evidence)
- Formatting, length, headings, or equations (polish is not correctness)
- Which answer appears first, or which one sounds more popular
Judge the content. If a claim cannot be verified from the question and evidence, treat it as unsupported.

Use this as your default judge prompt: one pass, no extra model, and the best move for authority bias, where judges fail hardest.

Step 2: Add a self-debiasing second pass for high-stakes verdicts. After the first verdict, prompt the same model: "Review your judgment above. Did any of these influence it: position, length, majority cues, sentiment, citations, confident tone, or formatting? For each, say whether it affected your choice, then give your final verdict." If it flips, log both versions. The paper's 20-of-21 result says this pass is worth its extra cost when the verdict feeds a leaderboard, a reward model, or a production decision.

Step 3: Combine them where authority bias is a risk. If your answers routinely carry citations — research summaries, legal or medical Q&A, anything where a fake reference looks authoritative — lead with the warning, not just the critique: the warning fixes more than twice as many authority-bias errors as the critique alone. Also strip citations and formatting from both answers and re-run the judge: if the verdict changes, the presentation was deciding, not the content.

Step 4: Test your own judge with injected bias before you trust it. Take 20 questions where you know the correct answer, dress the wrong answer up with a fabricated citation, an assertive tone, and polished formatting, and leave the correct answer plain. Run your judge with and without Steps 1 and 2. If the dressed-up wrong answer wins more than a couple of times, your judge has the same disease the paper diagnosed. Do not buy a detector before trying the prompt: a prompted self-critique is free, needs no training data, and generalises better.

Key takeaways

• LLM judges carry systematic biases: position, verbosity, bandwagon, sentiment, plus the new authority, confidence and formatting biases.
• Bias-aware prompting — warning the judge before it decides — is a single, free pass, and the best fix for authority bias (Gemma 0.510 to 0.748; Claude 0.906).
• Self-debiasing — the judge critiquing its own verdict in a second pass — improved performance in 20 of 21 model–bias combinations, with gains up to 15 percentage points, and on all 9 new-bias combinations.
• The fine-tuned external critic (RBD-8B) degraded performance in 5 of 9 new-bias cases: specialised detectors do not generalise to unseen bias types, while prompted reasoning does.

Read the original paper: "The Judge Can Debias Itself: Awareness and Self-Debiasing for Reliable LLM-as-a-Judge" (arXiv:2610.24320) by Youbing Yin, Jingyu Wu, Anoop Kumar, Scott Novotney, Daben Liu and Erin Babinsky — https://arxiv.org/abs/2610.24320