Case Studies

InnovationEval: Why Today's Best LLMs Can't Crack Problems Humans Solve in Their Sleep

Epoch AI's new benchmark tests whether AI can invent novel algorithms, not just implement known ones. Frontier models score single digits while human experts hit 60%, exposing a structural gap in how LLMs reason.

Font size

Epoch AI recently dropped a new benchmark called InnovationEval, built to measure whether AI can invent new algorithms rather than copy patterns it's already seen. The results are sobering: frontier models fail collectively, falling well short of even the human baseline.

The Hacker News thread on it pulled 46 points and 31 comments, hitting the front page. A recurring complaint in the discussion: models grab the problem and immediately start brute-forcing, getting stuck in local optima, with zero ability to "think sideways."

Worth unpacking.

What InnovationEval Actually Measures

Standard code benchmarks (HumanEval, LiveCodeBench, MBPP) test correctness. Given a problem in LeetCode style, pass the test cases and you win. Models crush these, with 96%, 97% scores commonplace.

But there's a hidden assumption baked in: the algorithm is already known. The model just translates an existing approach into code.

InnovationEval flips this. It tests algorithmic innovation: can the model figure out a novel computational strategy from scratch? The format looks like this:

  • A problem description (sometimes with a toy example)
  • A batch of input data (potentially thousands of test samples)
  • The task: produce a black-box function that returns the correct answer for each input
  • The function can inspect past samples, but ground-truth labels are hidden
  • The actual problem statement is obscured; the solver has to reverse-engineer what's being asked from data alone

This setup is critical. The model has to figure out what the problem even is, similar to interactive problems in competitive programming. You submit answers, get back correct/incorrect feedback, but no reference solution or hint about the approach.

Two concrete examples to ground this:

Example 1 (publicly released by Epoch): "Counting Rearrangements." Given a batch of operation sequences, predict the total count of some specific rearrangement. Human contestants, working for 3 hours, ran through 370+ samples and landed on an O(n log n) solution. Top humans hit 100% on the large test set.

Example 2: A packing problem, fit items into containers while minimizing some cost. Humans reach for greedy + heuristic search. Models default to brute-forcing all combinations.

Epoch recruited its test subjects from the competitive programming circuit: ICPC contestants, researchers, and dedicated competitors. Not casual users. Their baseline is the bar models need to clear.

Score Breakdown: Which Models Fell

Epoch ran a slate of frontier models. Approximate numbers from their 2025 report:

  • Human experts: 60% (average of top performers on large datasets)
  • GPT series (GPT-4, GPT-4o): roughly 2–8% pass rate across 4 problems of varying difficulty
  • Claude series: slightly above GPT, but under 15%
  • Gemini series: similar range

In other words, the strongest LLMs don't reach half of human performance.

In the HN comments, multiple users noted models falling into a "brute-force enumeration" mode. With no clear problem statement, models lean on "try every possible combination." That works on toy examples but collapses at scale (thousands of samples).

Another pattern flagged: models often get stuck at local optima. They find something passable and stop exploring. Which raises the next question: is this a structural limitation of LLMs, or just an engineering gap?

Why LLMs Struggle: Three Engineering Angles

1. Training data contains almost no "algorithm innovation" samples

Pretraining corpora come from the public web. The web has LeetCode solutions, papers, Stack Overflow answers. What it rarely has: records of humans innovating in real time, the kind of 3-hour whiteboard session where someone goes from confusion to insight.

Even when process logs exist, most are result-oriented ("we invented algorithm X") without the dead-ends, pivots, and backtracks that define real innovation. Models learn finished algorithms, never the birth of one.

2. Inference lacks systematic exploration strategies

LLM inference is autoregressive, one token after another, with no real backtracking. Chain-of-Thought and Self-Consistency improve "thinking one step further," but they're still walking a single chain.

For tasks demanding heavy exploration (innovation problems have huge solution spaces), you want Tree-of-Thought or structured search (MCTS, evolutionary algorithms). Some teams are already building Agent-style LLMs that run long-horizon exploration: modifying their own code, verifying their own results. But native frontier-model inference doesn't ship with this capability.

Side note: this "let the model explore actively" engineering approach overlaps with LangGraph-style state-machine-driven Agent design, using framework scaffolding to compensate for gaps in native reasoning.

3. The benchmark itself may be unfair to models

Worth giving some benefit of the doubt. InnovationEval has a hidden trap: the problem description is part of the prompt. What reads as intuitive to humans (we're good at semantic understanding) might be misinterpreted by a model when phrasing is ambiguous.

Epoch itself acknowledges in the paper that some problems are unfriendly to LLMs, particularly ones requiring specific mathematical-symbol intuition in domains where training data is sparse. That's a benchmark-design issue, not pure capability.

Why This Matters More Than It Looks

Algorithmic innovation is a recurring touchstone in AGI discussions. Products like OpenAI's o1/o3, Deep Research, and Devin are all bets on "can AI do long-chain reasoning plus autonomous exploration?" If models can't even crack competition-level algorithmic problems, the "AI replaces programmers" narrative needs a discount.

But don't get too pessimistic. For one, these problems were designed for human competitors: top performers still spend 3 hours. Testing models with a 30-second budget is fundamentally unfair.

Second, Agent-ified LLMs (equipped with search, tool calls, long-term memory) have huge headroom here. People have already proven this on benchmarks like ARC-AGI.

A more accurate framing: LLMs in their current native inference mode aren't there yet on these problems. Whether they'll ever be is a separate question.

Editorial Take

This might be revealing an "evaluation bottleneck" rather than a "capability bottleneck," something that the past two years of LLM score inflation have papered over. HumanEval, GSM8K, MMLU have all been driven to 95%+. ARC-AGI went from ~30% in 2023 to ~50% for some models in 2025. Benchmarks that actually differentiate frontier models from humans are getting rarer; most leaderboards differ by single-digit percentages.

InnovationEval fills a gap. It measures something humans can comfortably lead on, the kind of gap that actually matters. A 96% vs 97% spread on HumanEval is engineering-noise. A 60% vs 5% spread means the models haven't even entered the race.

My prediction: within three to five years, models will probably push InnovationEval scores to 70% or even 90%, through search + trial-and-error + tool calls, not through genuine algorithmic intuition. Looking at data and immediately sensing structure, landing key insights with zero trial-and-error: that capability is unlikely to emerge from autoregressive models in the near term. Benchmark designers who chase scores alone will fall back into the "score goes up ≠ equivalent capability" trap.

What's different this round: the test isn't "did you get it right," it's "could you even think of it." The former can be faked through memorization and pattern matching. The latter can't be brute-forced with scale. That's where InnovationEval actually hits the LLM pain point.


References:

Source: https://epoch.ai/publications/innovationeval