Amazon Science recently published a blog titled When LLM judges agree, should we believe them? — a question that sounds almost rhetorical until you realize the entire industry has been treating multi-model voting as ground truth.
The question hits at the root of a foundational assumption.
Original blog post: https://www.amazon.science/blog/when-llm-judges-agree-should-we-believe-them
Agreement Is Not Accuracy: The Core Question
Some background first. LLM-as-Judge has been one of the most overused evaluation methods of the past two years: let GPT-4, Claude, Gemini, and similar models score another model's output, then take the average or majority vote and call it your "automated evaluation result."
The logic sounds clean:
- Human annotation is too expensive
- If several strong models agree, the answer is probably right
But Amazon's research asks a straightforward question: when they all agree, are they actually correct?
The core finding isn't surprising, but it's humbling: when multiple LLM judges converge, their collective judgment can deviate systematically from ground truth on certain tasks. In other words — they can all be confidently wrong together.
This has significant implications for the current LLM evaluation ecosystem. RLHF pipelines, Reward Model training, and automated scoring on public benchmarks all embed "multi-model voting" at some stage. When the foundational assumption is questioned, everything built on top of it needs a second look.
How Amazon Tested It: Experimental Design
I haven't seen the full paper, but the methodology can be reconstructed from the blog abstract and HN discussion:
Multiple LLM judges score the same batch of samples, and researchers analyze two dimensions:
- Inter-judge agreement: How much do models A, B, and C agree with each other?
- Calibration: When all judges agree, how well does that consensus correlate with the actual correct answer?
Another key concept: bias propagation — if judges share training data sources, they may collectively fail on the same categories of errors.
Based on the public abstract, the experimental setup likely includes:
- Multiple commercial and open-source LLMs acting as judges
- Comparison of "accuracy when judges agree" vs. "accuracy when judges disagree"
- Coverage across multiple task types (QA, summarization, reasoning, etc.)
One HN commenter summed it up well: "Consensus among LLM judges ≠ ground truth. They can all be confidently wrong together."
Top Observations from the HN Discussion
The research made the Hacker News front page with 37 points and 20 comments (HN id 49699590). Modest engagement by HN standards, but the quality was high. Several upvoted observations stand out:
1. The Circularity Problem
"This study uses LLMs to evaluate LLM judges. How is that not circular reasoning?"
This comment drew significant agreement. Any study evaluating whether LLM judges are correct still needs some way to verify that correctness, and the most accessible tool is — you guessed it — another LLM. It's a chicken-and-egg problem that doesn't fully resolve.
2. Training Data Bias
Another frequent point: all these LLM judges are trained on similar web corpora, so they naturally make the same kinds of mistakes when encountering the same patterns. For trick questions — ones that look correct but are actually traps — multiple models may fall for it in unison.
3. The "Good Taste" Question
Some commenters went a level deeper: even if consensus ≠ correctness, it can still be useful for filtering obviously bad outputs. The question isn't "should we use LLM judges at all," but rather "in which pipelines can we use them, and how much should we trust them?"
Impact on Engineering Practice: Which Evaluation Pipelines Need Work
Concretely, which workflows does this research affect?
RLHF and Reward Model Training
A large share of the "preference data" in RLHF comes from LLM annotation. If that annotation carries systematic bias, the resulting Reward Model inherits that bias. If Amazon's conclusion holds, it's effectively telling every RLHF team: your model's "values" might be the output of a biased function.
Automated Benchmark Scoring
Many public leaderboards have already replaced human scoring with automated LLM judges. When all competing models are scored by the same panel of LLM judges, and those judges share biases, the rankings may reflect not model capability but ability to please the judges.
Practical Recommendations
My take: handle it in tiers.
| Scenario | Use LLM judge? | Notes |
|---|---|---|
| High-stakes decisions (medical, legal, financial) | ❌ Must be human | Non-negotiable |
| RLHF preference labeling | ⚠️ Use with caution | Pair with manual spot checks |
| Internal benchmarks during rapid iteration | ✅ Acceptable | Keep human sampling for validation |
| Public leaderboards | ⚠️ Use carefully | Hybrid scoring + publish inter-judge agreement metrics |
One engineering tip worth remembering: never look at "whether judges agree" alone — break down the confidence separately. Multiple judges agreeing at low confidence is more suspect than agreement at high confidence. Agreement is a necessary condition, not a sufficient one.
Editorial Thoughts
The real takeaway isn't "LLM judges can't be trusted" — anyone working in this space already knows that. I've been involved in several RLHF projects, and every team has the same recurring argument about "what proportion should be human-labeled."
The genuine value of Amazon's blog is this: it took a topic that everyone in the industry privately mutters about but nobody wants to put on the record, and turned it into formal research. A major company publicly acknowledging that its evaluation stack has problems is more significant than the specific finding.
A contrarian angle worth raising: rather than "LLM judges can't be trusted," the better question is what kind of evaluation system do we actually need?
The current evaluation paradigm has a fundamental flaw — it optimizes for "approaching the reference solution" instead of "actually solving the problem." LLM judges amplify this flaw, but the root cause isn't the judges themselves. The next generation of evaluation systems might require:
- Finer task-grained specialization (different judge combinations for different tasks)
- Consensus as a signal, not ground truth — treat it as a feature, not a label
- Adversarial samples designed specifically to test "judges collectively fail" scenarios
An honest caveat: when I read Amazon's blog, I had access to the abstract and not the methodology details. If the study itself has methodological weaknesses (e.g., sample selection bias), the strength of the conclusion should be discounted. But the direction is right — this is a problem the entire industry needs to confront.
One more note: 37 points on the HN front page counts as moderate-to-good engagement for a post. For a methodology-heavy blog, reaching the front page at all signals meaningful community interest.