Last week New Scientist dropped a story that deserves a closer look.
OpenAI had previously released a project aimed at using an LLM to help derive a proof of existence of smooth solutions to the Navier-Stokes equations—one of the seven Clay Millennium Prize Problems, so the stakes are real. The overall structure runs in two stages: Stage 1 has the LLM perform mathematical reasoning; Stage 2 translates the math into Lean formalization code. The failure happened not in the reasoning stage, but in the translation stage.
The story landed on the HN front page, and a common reaction in the comments was "LLM hallucinated again." It's more complicated than that.
What happened: how a proof is built in two stages
First, the structure of OpenAI's project, since all the discussion below depends on it.
[Input: mathematical proposition P]
↓
[Stage 1: mathematical reasoning]
→ LLM outputs an informal proof sketch (natural language + math notation)
↓
[Stage 2: math-to-code translation]
→ Translate the proof above into Lean code
→ Lean compiler typechecks, yielding a machine-verified proof
In theory, this is a clean design. Stage 1 lets the LLM do what it's good at—free-form reasoning and pattern matching. Stage 2 uses Lean, a dependently typed language, as a final gatekeeper: only typechecked code counts.
The problem sits in Stage 2.
The mathematical sketch produced in Stage 1 was reviewed by domain experts and considered reasonable—the logical chain roughly holds. The errors crept in during the process of writing it up as Lean code. In other words, the LLM's mathematical reasoning was basically correct, but during translation, certain semantics got quietly swapped out.
These errors are sneakier than outright reasoning failures. Lean's typechecker will pass—as long as the syntax is valid and types match, Lean lets it through. It won't tell you "the proposition this code expresses isn't the same as the proposition you meant to prove."
What the errors look like: not broken reasoning, warped translation
New Scientist's coverage flagged a few typical problem patterns. Here's my take on the mistakes LLMs most commonly make in formalization settings.
1. Symbol-semantics drift
The same symbol carries slightly different meanings in different contexts. Take a quantifier chain like ∀ε>0, ∃δ>0: ...: a mathematician writes it clearly, but the LLM may flip the quantifier order or bind the existential to the wrong variable scope.
The corresponding Lean code might look like:
-- What the mathematician meant:
-- ∀ ε > 0, ∃ δ > 0, ...
-- What the LLM actually wrote:
∃ δ > 0, ∀ ε > 0, ... -- quantifiers in the wrong order
Lean typechecks it. The mathematical meaning is completely different.
2. Dropped quantifiers
The sneakiest kind. The LLM mentions a universal quantifier during reasoning, then drops it during translation because it feels "obvious." The result: Lean proves a weaker statement than the one originally claimed.
3. Continuous-to-discrete approximations, silently ignored
Navier-Stokes is a PDE, and the core work happens in estimates over infinite-dimensional function spaces. The LLM might reason that "by the Sobolev embedding, ...," which sounds right, but the Sobolev embedding has specific dimension and boundary conditions. If the LLM doesn't write those preconditions out explicitly during translation, the proposition Lean verifies is not the original one.
4. Type-level semantic substitution
More technical: the function space on ℝ in Lean might end up as some subtype, or Continuous might become ContinuousOn—one word different, different mathematical meaning.
The common thread: when reviewing LLM-generated code, a human who doesn't scrutinize every quantifier and type qualifier will subconsciously feel "this looks right." That is the most dangerous thing about formalization tasks.
Why this case matters: the boundary of L4 autonomous reasoning
OpenAI has an internal grading scheme for its models' reasoning capabilities (L1 through L5, roughly tracking task difficulty and reliability requirements). This grading appears in their safety/capability evaluation documentation. The Navier-Stokes task falls around L4—long-horizon, formal, requiring strict verification of mathematical reasoning.
L4 is a delicate tier. Below L5 (AGI-level), but well above L3 (short tasks, single domain). Claiming to have crossed L4 means the model can handle long-duration work that requires continuous logical discipline. Formal proofs are a textbook L4 task.
The signal from this incident is clear: the L4 bar hasn't been cleared yet.
And this isn't a "close but no cigar" situation. The fact that mathematical reasoning itself ran through shows the model genuinely has substantial math ability. But between "can produce a decent proof sketch" and "can fully and precisely translate that sketch into formal code," there's a semantic gap. That gap isn't filled automatically by scaling the model or training on more rounds of data—it fundamentally requires the model to have precise, exact understanding of every mathematical object, every symbol, every quantifier, rather than the statistical "looks right."
The formalization community is most alert to this because they've seen "looks right" fail too many times. Systems like Lean, Coq, and Isabelle are designed on the philosophy of "trust no one, trust only the typechecker." But a typechecker can only guarantee that the code you wrote matches the meaning you expressed—it cannot guarantee that the meaning you expressed is what you originally meant.
Impact on the Lean/Coq community and AI4Math
The impact on the formalization community isn't that "OpenAI was wrong," but that this provides a concrete, dissectible case.
The dominant AI4Math narrative over the past few years has been:
- LLM produces a proof sketch
- Human mathematician reviews it
- Lean compiler typechecks it
- Three layers of safety, all good
What this incident exposes: the sketch-review layer is almost impossible for humans to perform on the math-to-code translation.
Why? Because Lean code isn't written for humans to read—it's written for the compiler. An experienced Lean programmer can spot the difference between ∃ δ, ∀ ε and ∀ ε, ∃ δ, but when a proof runs tens of thousands of lines, the probability of catching that kind of detail-level error drops fast.
In other words, the human + LLM + Lean combo isn't meaningfully better than a human alone, at least in the translation stage.
Expect a few reactions from the Lean community:
- Stricter provenance tagging for LLM-generated Lean code—distinguishing human-written from machine-written.
- New auxiliary tools that perform "semantic-layer verification" on top of typechecking—for instance, checking whether the actual mathematical content of a proof matches the claimed proposition.
- Tighter review standards for AI4Math papers—reviewers may start demanding accompanying Lean code, not just proof sketches.
The second item is worth watching closely. Semantic-diff-style tools may move from research prototype to standard equipment in LLM translation scenarios.
Editor's take
The real signal here isn't "AI made another mistake." It's that when each substep looks fine on its own, an LLM's cross-step semantics drift silently. Drift is far more dangerous than outright failure—when something fails, you can catch it. But with drift, the code typechecks, the natural language reads smoothly, the quantifiers look like they're in the right place, and there's no obvious signal telling you "here is the error," until eventually a mathematician or a concrete counterexample blows it open.
Unlike the optimistic AI4Math narrative of the past few years, this incident tears open a crack: LLMs are genuinely strong at the local level of mathematical reasoning (the fact that Stage 1 didn't fail is the evidence). But the gap from local to global, from reasoning to formalization, may be larger than most people think. "Human + LLM + formal system" has been marketed as triple insurance, but in the translation step, the human provides no insurance at all—because the code isn't meant for humans to read.
A contrarian prediction: in the next 12–18 months, the Lean/Coq community will see "semantic verification tools" built specifically for LLM-generated code, functioning like linters on top of the compiler. This will become standard practice not because anyone wants it, but because there's no other choice. The formalization community will evolve from "we trust no one, only the typechecker" to "we trust no one, not even the typechecker, not even the LLM; we trust only semantic-layer diff."
One unknown: the actual impact of this on OpenAI's internal model capabilities is hard for outsiders to judge. New Scientist's reporting is based on public materials, and there's no information on whether OpenAI has already caught these issues internally or how far along any fixes are. If they've spotted them but chosen not to disclose, that would actually be an honest signal—meaning they themselves know the L4 bar hasn't been crossed.
References: