FrontierHarness hit the front page of Show HN this week with 36 points and 11 comments. The premise is straightforward: run nine different agent harnesses against the same model and compare cost per pass across each one.
The result was surprising. The most expensive harness cost 17 times as much as the cheapest.
Same model. Same task set. Nearly all the variance came from the harness itself: how its tool-use loop schedules tokens, how it trims context, how it coordinates multi-agent workflows. The framework layer isn't just a wrapper. It's an active token-burning middleman.
Where the 17x Gap Comes From
FrontierHarness breaks out cost per pass across harnesses. Same prompt, same underlying model, nine harnesses, and the spread between the top and the bottom runs an order of magnitude.
A useful intuition: swapping models rarely saves much. GPT-4o vs Claude 3.5 Sonnet might be a 2–3x difference at most. But pick the wrong harness and you're looking at 17x. Most agent projects' billing pressure comes from the framework, not the model.
The harder pill to swallow is that harness cost differences are structural. You can't close the gap with better prompting.
Why Harnesses Burn So Many Tokens
Break it down and the expensive harnesses tend to waste tokens in the same places:
- Tool-use loop retries. A failed tool call can trigger 2–3 automatic retries. Each retry stuffs the full conversation history back into context. One failed call costs 2–3x the normal path.
- Unbounded context accumulation. After a few turns, the context fills with early tool outputs, error logs, and retry messages. Many harnesses don't trim by default and feed the entire history into the next turn.
- Multi-agent coordination overhead. Frameworks like CrewAI and AutoGen pass messages between agents and coordinate plans. That coordination cost typically lands on the user's bill.
- System prompt bloat. Some frameworks ship with default system prompts longer than anything you'd write yourself. Every token counts at input pricing.
Take LangGraph. Its state-graph design naturally supports fine-grained control. You can explicitly trim state and cap retries. The tradeoff is more glue code for the developer.
CrewAI is easier to pick up. Add a decorator and you're running. But its default agent-collaboration pattern aggregates all agents' history, so context balloons on long tasks.
AutoGen sits in the middle. GroupChat is flexible, but once you turn on multi-agent negotiation, the invisible "internal traffic" chews through tokens.
Expensive isn't a bug. Expensive harnesses often do more for you. But most developers don't know what they're paying for until the invoice teaches them.
How Reliable Is the Benchmark
FrontierHarness is a useful starting point, but a few questions after reading the project page:
- Which nine harnesses? The specific list significantly shapes the conclusion. If they're all research-oriented (ReAct, Reflexion, etc.) versus a mix that includes production frameworks like LangGraph and CrewAI, the result means very different things.
- Single-turn or multi-turn tasks? Single-turn Q&A shows minimal harness-to-harness variance. Multi-turn, cross-tool, long-context tasks are where the gap widens. If the task design is too simple, the 17x figure is likely an underestimate.
- Are token counts consistent? Some harnesses count internal message-routing tokens; others only count user-facing tokens. Same "cost," potentially 20% different depending on accounting.
The problem with benchmarks is that they usually tell you who's cheap without telling you why. A $0.01-per-pass harness might be cheap because it doesn't do what you actually need (auto-retry, context compression, multi-agent coordination), not because it's "efficient."
What Developers Can Do Today
Whether or not FrontierHarness's methodology is airtight, a few checks are worth running right now:
1. Calculate your own cost per task. Run your harness 100 times, log the total spend, divide by task count. Most agent projects' cost per task is far higher than developers assume, because retries, context accumulation, and bloated system prompts are invisible in normal logs.
2. Disable unnecessary tool loops. Many harnesses ship with "smart retry" enabled by default. Check the config and drop retry attempts from 3 to 1. That alone can eliminate a lot of redundant tokens.
3. Audit system prompt length. Print your system prompt and look at it. Often you'll find hundreds of lines of framework-default documentation crammed in. If your actual instructions are 200 words but the prompt is 2,000, every turn is paying for 1,800 words you didn't write.
FrontierHarness itself works as a self-test entry point (frontierharness.org). Feed in your own prompt and see where your current setup lands on the cost distribution.
Editorial Take
The cost premium in agent harnesses isn't a bug. It's a feature. But that feature's pricing is opaque to developers.
This is different from past LLM framework debates (LangChain vs LlamaIndex and the like). The harness directly burns cash, while ORM-style abstractions stay roughly stable. Writing extra glue code doesn't make your database query more expensive. Pick the wrong harness and your bill doubles that month.
My prediction: in the next 6–12 months, a wave of "harness profiler" tools will appear, focused on cost tracing and token breakdown. Think of the current frontend bundle analyzer: you deploy and then discover that 80% of the bundle is unused dependencies. The harness version of that discovery is: you deploy and then realize 80% of your tokens went to the framework itself.
The deeper signal: agent framework competition will shift from "how many features" to "how much capability per dollar." Whoever completes more tasks within a $1 budget wins. Feature-stacking frameworks (the CrewAI model) will get harder to sell.
One uncertainty: I can't independently reproduce FrontierHarness's benchmark. The specific list of nine harnesses and the task design matter a lot for the conclusion. If anyone has run the original benchmark, drop your numbers in the comments.
Source: FrontierHarness project page