Frameworks

Harness, Not the Model, Explains 90% of Coding Agent Benchmark Gains

A controlled study called HarnessTax shows that the wrapper layer—prompts, tool definitions, and feedback loops—accounts for most of the performance gap between coding agents. Treat those leaderboard scores with skepticism.

Font size

Over the past year, vendors have been jockeying for position on coding agent leaderboards. SWE-bench Verified scores climbed from 30% to over 60%, which looks like a model improvement story. But the HarnessTax project says: not so fast—the harness (wrapper layer) likely contributes far more to those numbers than you'd think.

The Problem: Models Got Better, Agents Didn't (Much)

I've used several coding agents hands-on. GPT-5, Claude 4.5, and DeepSeek V3.2 feel better than the previous generation, but not 20 points-of-SWE-bench better. On real tasks, the gap feels closer to 5 points.

HarnessTax's hypothesis is straightforward: the differences come from prompt scaffolding, tool definitions, and error-retry mechanisms—not from the base model. Swap out the wrapper and you can move the score by 30+ points on the same underlying model.

The project homepage summarizes it this way—the study systematically separates the impact of the harness (prompt scaffolding, tools, feedback loops) from the base model on coding agent performance.

Plain English: when you run a benchmark, you're measuring the agent, not the model. But everyone is using agent scores to rank models.

How the Experiment Works: Stripping the Harness with Controlled Variables

The core method is controlling variables. Fix the base model (say, a specific version of GPT-4o or Claude 3.5 Sonnet), then vary only the harness: prompt templates, JSON schemas for tool calls, retry counts, whether prompts change between retries, and context window management strategies.

At scale: the study ran multiple mainstream coding benchmarks—SWE-bench, LiveCodeBench, HumanEval, and similar. For each benchmark, they crossed the same set of base models with multiple harness configurations.

Think of it as A/B testing taken to its industrial conclusion. Rather than changing one variable at a time, they decomposed the harness into independent dimensions (prompt structure, tool API design, feedback loop mechanism) and measured each dimension's contribution separately.

# Simplified version of the experimental design
from dataclasses import dataclass
from typing import List

@dataclass
class HarnessConfig:
    prompt_template: str      # prompt structure
    tool_schema: dict         # tool definitions
    retry_strategy: str       # error retry mechanism
    context_management: str   # context window strategy

# Fix base_model, vary only the harness dimensions
experiments = [
    HarnessConfig("minimal", basic_tools, "none", "truncate"),
    HarnessConfig("structured", rich_tools, "adaptive", "sliding_window"),
    # ... dozens of combinations
]

for config in experiments:
    score = run_benchmark(base_model, config)
    record(config, score)

Results: Differences Big Enough to Mislead Investment Decisions

The project homepage states it plainly: harness-level differences can cause significant—even misleading—fluctuations in agent performance.

On Hacker News, the project reached 62 points with 17 comments (HN item id 49733726—figures as of review time and subject to change). One highly upvoted comment in the HN thread put it well: nobody had systematically answered how much of the score gap comes from the model versus the harness. That captures what a lot of people were wondering.

From an investment angle: a company valued in the billions tells a story of "our model is better than OpenAI's," but the score difference might be 80% from a carefully tuned harness rather than the model weights. That's a fragile foundation for a valuation.

From an engineering angle: you spend two months tweaking prompts, score jumps 15 points, and you credit the new base model—when really you just got the harness right. Next time the base model upgrades, those 15 points may vanish.

Leaderboards in the Crossfire: Does That Top-1 Still Count?

The benchmarks called out are the frequently cited ones:

  • SWE-bench Verified: now considered the coding agent gold standard. But the harness's share of each score is unclear.
  • LiveCodeBench: leans toward contest-style problems, so harness influence is smaller—but still non-negligible.
  • HumanEval: too old, mostly retired, though it still shows up in some papers.

How should readers treat these numbers? My suggestions:

  1. Relative scores beat absolute scores. If Model A beats Model B by 5 points under the same harness, that 5 points is far more credible.
  2. Look for open-source reproduction. If a leaderboard score only exists as a vendor run with no open-sourced code, apply a 30% discount immediately.
  3. Look at real tasks. A benchmark is a benchmark; what matters is what happens on your codebase.

Practical: Building Your Own Agent Without the Common Pitfalls

If harness really matters this much, how should engineering priorities be ordered?

Phase 1: Get the harness right

  • Structure your prompts (tell the model explicitly what to do at each step)
  • Keep tool definitions atomic (don't make one function do ten things)
  • Make error feedback carry concrete context (not just "error: NoneType"—specify "when you called X, the Y field was missing")

Phase 2: Only then consider switching models

  • Test different base models under a fixed harness
  • Less than 5 points difference—not worth switching (cost, latency, compliance risk)
  • More than 10 points difference—worth considering

Decision framework (my own):

SignalAction
Same harness, different models, < 5% gapDon't switch models; keep tuning the harness
Same harness, different models, 5–15% gapEvaluate switching cost (latency, price)
Same harness, different models, > 15% gapSwitch, but re-tune the harness afterward
Different harnesses, same model, > 20% gapThe harness is the real bottleneck

The last row is the key one. If swapping a prompt template gains you 20 points, fine-tuning model weights is the worst ROI you can chase.

Editorial Take

The point isn't the trivial observation that "harness matters." It's that this work exposes the statistical illusion in current coding agent evaluation. We use benchmark scores to rank models, and rankings to value companies—but nobody had systematically calculated how much of the score is the model itself versus the wrapper engineering. HarnessTax is the first to do it.

My prediction: within the next six months, a wave of "harness benchmarking" will emerge. Teams will build benchmarks specifically for agent harnesses, much like today's prompt engineering evaluations. The value of frameworks like LangGraph and AutoGen will be reassessed—they're selling harnesses, not models.

How this differs from past work: earlier papers discussed prompt engineering's importance, but those were empirical, case-study style. HarnessTax is the first to use controlled experiments to systematically quantify the harness's contribution. Moving from "I think prompts matter" to "prompts contribute 30 more points than the model" is a leap from anecdote to evidence.

Where I'm uncertain: can the harness contribution be extrapolated to non-coding scenarios? Web agents, GUI agents, research agents. Intuitively, those scenarios have more complex harnesses (more tools, longer decision chains), so the harness share might be even higher. But that needs experimental validation. If the extrapolation holds, the entire agent evaluation system needs to be rewritten.


Sources:

Source: https://harnesstax.github.io/