MaxxBench did not simply repeat DeepSWE's results. It asked a more fundamental question: “Nice DeepSWE score. But is it true?” The dispute is not only about rankings, but about whether leaderboards can measure real, reproducible software engineering ability.
Before Celebrating a High Score
Another impressive result has appeared on the code Agent leaderboard.
MaxxBench then published a research article titled Why MaxxBench. Rather than highlighting DeepSWE's score, it opened with a restrained but consequential question:
“Nice DeepSWE score. But is it true?”
It does not allege that DeepSWE cheated or declare its score invalid. Instead, it asks a deeper question: Did the Agent actually complete the task?
MaxxBench evaluates the performance of AI software engineering agents. Its public summary focuses on the credibility and reproducibility of DeepSWE's high score. The original research is available here:
- Research article: https://maxxbench.unlap.ai/research/why-maxxbench/
- Hacker News discussion: https://news.ycombinator.com/item?id=50406222
The research reached the Hacker News front page, earning 77 points and 15 comments.
Developers clearly care less about who finished first than about whether a leaderboard of this kind can still distinguish Agents' actual capabilities.
The boundary needs to be clear. The available summary does not disclose DeepSWE's exact score, task sample, number of reproduction attempts, or environment configuration. It therefore does not prove that the result was fabricated.
But “no evidence of fabrication” is not the same as “the high score has been proven reliable.”
Are Benchmarks Measuring Problem Solving or Engineering?
Code Agent benchmarks typically provide a software repository, an Issue, and a set of tests. The Agent modifies the code and submits its work. If the tests pass, the task is considered complete.
This process works well for automated evaluation. But what it measures may be “problem solving” rather than a complete form of “engineering.”
At least four layers need to be separated.
The first is requirement comprehension.
An Issue might say, “Fix user export failures.” An Agent may only make the existing tests pass without handling empty files, oversized files, permission errors, or unexpected encodings. The absence of test coverage does not mean the requirement has been met.
The second is code correctness.
Passing public tests only demonstrates that public examples satisfy the assertions. Hidden tests may cover edge cases, concurrency problems, or regressions in existing functionality.
The third is maintainability.
To pass the tests, an Agent may duplicate logic, bypass validation, hard-code inputs, or implement the fix at the wrong abstraction layer. As new requirements arrive, maintenance costs will continue to climb.
The fourth is delivery readiness.
Code that compiles does not necessarily deploy. Deployable code does not necessarily support troubleshooting. A local script that runs successfully does not necessarily remain stable across CI systems, containers, and dependency version changes.
Real software engineering is not just “making the tests green.” It involves reading code, identifying boundaries, modifying multiple modules, validating side effects, and leaving a change that others can understand.
Agent benchmarks should therefore report at least two things:
- Whether the task passes the tests
- Whether the task fulfills the requirements
With only the first number, the leaderboard resembles a code repair competition. That has value, but it cannot directly be presented as a ranking of real software engineering ability.
Reproducibility Gaps Matter More Than Scores
Evaluating an Agent run requires fixing at least the following variables:
| Variable | Why it matters |
|---|---|
| Model and version | API models may update silently, so two models with the same name may no longer use the same weights. |
| Prompt and context | Agent behavior may change with the system prompt. |
| Tool permissions | Network access, full-repository read access, and permission to run arbitrary commands can materially affect the result. |
| Tool implementations | Differences among shell, editor, search, and testing tools can alter the trajectory. |
| Repository snapshot | Once the main branch changes, both the Issue and its dependencies may differ. |
| Dependency environment | Versions of Python, Node.js, compilers, and system packages need to be pinned. |
| Scoring script | The completeness of the tests and the accuracy of their assertions determine the final score. |
| Retries and budget | Rules must specify the number of reruns, token limits, time limits, and costs. |
This raises a practical problem.
Many Agent leaderboards publish task sets and final scores but not complete execution records. Others see a “90,” with no visibility into the model snapshot, tools invoked, files changed, or number of test runs.
This is not a matter of mystique. It is a flaw in experimental design.
A result that cannot be reproduced cannot support meaningful comparisons.
A more reliable approach is to save every run as a structured record. At minimum, it should include the model configuration, container image digest, repository commit, tool list, budget limits, final diff, and scoring logs.
For example, an evaluator could generate a manifest for every run:
{
"task_id": "export-empty-file",
"repository_commit": "0123456789abcdef",
"model": "provider/model@2026-08-01",
"system_prompt_sha256": "<64位十六进制摘要>",
"docker_digest": "sha256:<镜像摘要>",
"tool_permissions": ["read_repo", "edit_repo", "run_test"],
"max_steps": 50,
"max_cost_usd": 5.0,
"score_script_sha256": "<64位十六进制摘要>"
}
This does not require every Agent to expose proprietary internal data. Publishing a reproducible subset would still be more useful than releasing only an overall leaderboard.
Why the HN Discussion Focused on Benchmark Failure
The research reached the Hacker News front page, earning 77 points and 15 comments.
The discussion did not focus on which model won or lost. It focused on the failure of benchmarks.
The reason is not difficult to understand.
Developers deal with GitHub Issues, CI failures, and production traces every day. For them, an Agent solving a benchmark task and an Agent handling a real repository are two different things.
Public tests also create a familiar problem: the tests themselves become the optimization target.
Agents learn the shape of the evaluator. Hidden tests are harder to target than public tests, but they are still not equivalent to requirements acceptance. The larger and more complete the repository and test suite, the closer a passing test result is to a correct implementation. For small tasks with sparse edge cases, that gap can be substantial.
Another problem is that task difficulty changes along with Agent capability.
An Agent may solve an Issue today but use a different search strategy, backend model, or tool wrapper tomorrow. The leaderboard appears to compare models, but it may actually compare combinations of models, tools, scaffolding, and budgets.
This does not mean benchmarks have no value.
Without shared tasks, Agent developers would struggle even to identify a direction. Benchmarks are still necessary. They should, however, be treated as experimental workbenches rather than professional certification.
Fixing Tasks Is Not the Same as Replacing Leaderboards
What should Agent evaluation do next?
First, retain hidden tests, but do not make them the sole judge.
Hidden tests can reduce cheating against public assertions. Human acceptance must examine requirement coverage, code quality, and side effects. The two should be combined rather than treated as substitutes.
Second, incorporate historical code repositories.
Tasks should ideally come from real Issues and include multiple commits, related pull requests, regression tests, and behavior from earlier versions. Agents would then need to work with a system that has a timeline, not merely a current directory.
Third, record the complete trajectory.
Beyond the final diff, evaluations should preserve tool calls, test output, failed attempts, and decision logs. This makes it possible to distinguish an Agent that identified and fixed the root cause from one that happened to satisfy the tests.
Fourth, publish results across multiple runs.
A single successful run is easily affected by randomness and environmental variation. Evaluations should report at least the number of runs, success rate, and confidence interval rather than only the best result.
Fifth, break the score into components.
Test pass rate, human acceptance rate, average cost, average duration, number of tool calls, and number of invalid edits should be reported separately. Developers can then determine whether an Agent is “more accurate” or merely “more expensive.”
The final leaderboard should preferably take the form of task-level reports:
Task: fix-export-empty-file
Automated tests: passed
Manual acceptance: failed
Reason: CSV header missing for empty input
Invalid edits: 2
Cost: $1.42
Elapsed: 4m 18s
Such reports are not especially glamorous. They answer the questions engineering teams actually care about.
Editor's Perspective
The significance of this episode is not simply DeepSWE's score. AI Agents are now competing on real software engineering tasks, so the central question has shifted from whether a model can write a good explanation to whether its engineering process can be verified.
Model launch posts usually include carefully selected examples. Agent leaderboards, by contrast, may encourage readers to mistake a single execution record for a stable capability. The difference lies in the nature of the output: an Agent's answer is an entire chain of operations, and repository state, tool permissions, and retry strategies all affect the final result.
Because the available public summary lacks the exact score, sample size, number of reproduction attempts, and environment configuration, DeepSWE's leaderboard performance cannot be declared fabricated. The next phase of Agent benchmarking will not be won by publishing another leaderboard with a higher aggregate score. It will be won by making tasks, trajectories, and failure explanations durable enough for third parties to rerun.