For the past two years, AI coding benchmarks have mostly answered one question: can the model fix the bug?
Fix it correctly, and the model ranks high. Fix it fast and cheaply, and it gets even more praise. SWE-bench Verified, DeepSWE, and SWE-bench Live, the three most cited leaderboards, all center on two metrics: solve rate and cost.
One dimension has gone unmeasured: how much unneeded code does the AI leave behind after fixing the bug?
Parsimony was built for exactly this gap. It recently hit the front page there, driven by a single idea: instead of counting lines, count net coding units.
Existing benchmarks are missing a dimension
SWE-bench Verified is the most cited AI coding leaderboard today. OpenAI, Anthropic, and Google all publish Verified scores with every new model release. DeepSWE comes from Cognition (the Devin team), and SWE-bench Live is a rolling version maintained by Princeton.
What do all three measure? Pass rate. Token cost. None of them look at how much junk code the model quietly pushes into the repo.
This happens in real engineering every day. I once used Claude Code on a Next.js project. The model scored full marks on solve rate, but the PR added 800 lines, half of which were imports for helpers that were never called, plus a block of logic copy-pasted from another file and tweaked. The functionality was fine. The person inheriting that code would not be happy next week.
Current benchmarks cannot detect this at all.
How Parsimony measures "net code"
The methodology page contains a line I think is the smartest part of the project:
Deleted code is subtracted, so it measures net units.
Parsimony does not just count what the AI added. It subtracts what the AI deleted. The final score is net coding units, not raw line count.
Why not just count lines? Because lines are trivially gameable. A model can pad a function with blank lines and meaningless comments, and LoC goes right up. Every programmer knows LoC is a terrible metric.
Parsimony instead counts coding units, broken into four categories:
- statements
- calls (function calls)
- names (variables, classes, identifiers)
- literals (strings and numbers)
These are pulled from the actual syntax tree, real code density, not formatting tricks that produce fake lines.
To game the score, an AI would have to produce meaningful syntax nodes, which is effectively just writing code. The closer a metric sits to the semantic layer, the smaller the gaming surface. This direction aligns with Simon Willison's earlier writing about "evaluating intent, not just output."
The author ran GPT-6 and Claude 5.5 himself
Parsimony's data did not come from a paper.
The project page states plainly that for GPT-6 and Claude 5.5, no public patches were available, so the author ran the models himself to fill in the data.
This detail deserves attention. Anthropic and OpenAI now ship new models faster than third-party benchmarks can keep up. SWE-bench's patch datasets are frozen months in advance. When a new model drops, the community either waits for official data or runs it themselves. Parsimony's author chose the second path. He acknowledges his numbers may diverge from vendor-internal results, but at least the leaderboard does not sit empty while everyone waits.
For an open-source evaluation project, that willingness to do the work matters.
What this means for long-running projects
Code bloat is a slow disease.
A PR that hits 100% solve rate today will hurt the person who inherits it three months from now. The extra 800 lines become:
- "Looks useful" functions nobody dares delete
- Copy-pasted logic from older parts of the codebase
console.logstatements and debug branches left over from investigation- Temporary fixes that become permanent
Each one is technical debt.
Parsimony's value is turning "future maintenance cost" into a number that can be compared today. Engineering teams picking AI coding tools now have one more scoring column: the tool has to fix bugs and fix them cleanly.
The logic mirrors what happened with cloud providers and "pay-per-effective-query" pricing. Early "pay-per-call" billing rewarded waste. Later "pay-per-result" pricing made optimization a competitive advantage. AI coding tools will reach the same point eventually.
Editorial thoughts
Every AI coding benchmark right now is racing on solve rate.
This is still exam thinking: get the right answer, ignore how ugly it looks. Real engineering is not an exam. The fact that longer code is harder to maintain has been taught in software engineering classes for decades, yet it still has not made it into a benchmark. That is hard to justify.
Parsimony is a start, and its limits are clear: it operates at the single-PR level. Cross-file effects are not considered (an AI that touches five files, deleting 20 lines from each, looks like net zero, but the module boundaries are already broken). Readability is also off the table (bad variable names, namespace pollution, things coding units cannot catch).
The direction is right, though.
An open question: if Anthropic and OpenAI added net code volume to their official rankings, what would change across the industry? Would model vendors add a "code conciseness" reward during training? Would a new class of models emerge specifically optimized to game this metric? Would it become yet another arms race?
I cannot predict that, but one thing is certain: once official benchmarks start penalizing code bloat, AI coding prompt work will shift from "how do I fix the bug" to "how do I write less code." At that point, prompt engineers may need to start reading Clean Code.
Project: https://parsimony.lasu.dev/
Methodology: Counts four categories of coding units (statements, calls, names, literals); deleted code is subtracted from the total.