FrontierHarness Eval v1.1 analyzed 27,280 coding agent runs and found that 64% of inference spending went to tasks that ultimately failed. For teams deploying AI Agents, this means success rate alone is not enough. Retries, unproductive loops, and tool calls also need to be accounted for.
64% of the Inference Bill Comes from Failures
An AI Agent's inference costs do not end when the model produces its first incorrect response.
Failed tasks often follow a familiar path:
- The Agent generates an initial solution but misses a constraint.
- It calls a tool and receives the result, but does not revise its plan.
- It repeats checks or regenerates output along the same path.
- A retry triggers another round of token consumption and tool-call quotas.
- The task still goes unfinished.
Each step may look reasonable in isolation. Across a large batch of tasks, however, unproductive loops scale with demand.
Failed runs accounted for 64% of inference spend, meaning that nearly two-thirds of inference spending produced no successful delivery. Possible causes include selecting the wrong model, excessive context, and execution strategies that break down.
This is also what a final pass rate can hide. Two models may each achieve a 70% success rate, but one stops after succeeding on the first attempt while the other requires five consecutive repairs to reach the same result. Their production costs are not equal despite having the same success rate.
A High Success Rate Does Not Mean Low Cost
Traditional coding agent benchmarks tend to focus on a few outcomes:
- Whether the submission passes unit tests;
- Whether the final submission runs;
- Whether human reviewers approve it;
- Whether it completes within the time limit.
These metrics remain useful, but they do not capture the economics of running an Agent.
I would record five types of data for every run:
| Metric | Question it answers |
|---|---|
| Success rate | What percentage of tasks are completed correctly? |
| Total tokens | How much context does the Agent consume for one task? |
| Tool calls | How often are search, terminal, editor, and external API tools called? |
| Retries | How much do errors amplify through repeated attempts? |
| wall-clock time | How long does the task actually take? |
Token usage and tool-call counts should be reviewed together. One Agent may produce lengthy analysis while using tools sparingly. Another may generate relatively little text but repeatedly launches tests, reads logs, and modifies files. The latter often better reflects the real burden on a production bill.
Wall-clock time also matters. A 20-minute wait may be acceptable for an offline development task, but the same delay in an online service directly affects user experience and timeout costs.
What Changed in FrontierHarness Eval v1.1
This release includes FrontierHarness Eval v1.1 for evaluating coding agents.
The important change is not simply a larger set of tasks. Evaluation now extends beyond the final answer to the full execution process.
For a coding agent, the final code records only the last page of the story. Evaluation also needs to reveal:
- How many times the Agent planned;
- Which tool calls returned errors;
- Whether it repeated identical operations;
- Whether it made effective adjustments after a failure;
- How inference costs were distributed before success.
This allows a test failure to be broken down further. It might reflect incorrect implementation logic, failure to read the project structure correctly, or an Agent trapped in an unproductive loop after a test failed.
For the original introduction, see FrontierHarness Eval v1.1.
The article also reached the Hacker News front page. At the time, the page showed 8 points and 2 comments. The discussion is available here: Hacker News discussion.
The limited size of that discussion does not diminish the practical value of the data for Agent teams. For teams using Claude Code, Cursor, or other coding agents, failed execution paths have already entered their API bills. They are simply not usually tracked separately.
Account for Failed Paths in Production Bills
The following metrics can be included directly in an evaluation report.
1. Cost per Success
The basic calculation is:
successful_cost = total_cost / successful_runs
This is closer to actual business spending than average cost per run because value is only realized when a task succeeds.
If Model A costs an average of $1 per run and has a 50% success rate, its cost per success is $2. Model B costs an average of $1.20 per run and has an 80% success rate, giving it a cost per success of $1.50. The second model appears more expensive but is actually cheaper.
2. Average Inference Used Before Failure
Group runs by outcome and calculate:
average_inference_before_failure = (
sum(run.inference_cost for run in failed_runs) / len(failed_runs)
)
This metric helps isolate problems. If failed tasks consume few tokens, the Agent may be giving up early. High token consumption before failure is more likely to indicate planning drift, context growth, or repeated debugging.
3. Retry Amplification Factor
A simple version is:
retry_amplification = total_runs / successful_runs
If an Agent succeeds 100 times after 180 total runs, the factor is 1.8. Each successful task required an average of 0.8 additional failed or retried executions.
A more detailed approach separates failures by cause, including test failures, tool timeouts, context-limit violations, and permission errors. Each cause requires a different repair strategy.
4. Failed Tasks' Contribution to the Total Budget
failed_budget_share = failed_inference_spend / total_inference_spend
The result in this study was 64%.
If that share continues to rise, teams should look beyond model routing and also check:
- Whether the Agent is receiving excessive irrelevant context;
- Whether tool results lack structured error information;
- Whether retries are repeating the same strategy;
- Whether the Agent is allowed to loop on identical tool calls;
- Whether simpler tasks should use a smaller model;
- Whether verifiable checks can run before expensive inference.
One direct control is to set a repetition threshold for identical tool arguments and error results. For example, after the same file read fails twice, require the Agent to change its approach rather than retry unchanged. The appropriate threshold must be tested in the project's environment and should not be copied blindly.
Production monitoring can establish cost baselines by task type:
def is_cost_anomaly(cost, task_type, baseline):
median, mad = baseline[task_type]
return mad > 0 and cost > median + 3 * mad
This uses the median and MAD rather than the mean and standard deviation because a small number of unusually expensive failures can pull the mean upward.
Editorial Perspective
The 27,280 runs provide a more production-relevant view of costs than a single benchmark because they expose failed execution paths, not because of sample size alone. Real systems do not retain only successful cases. Numerous failures, repeated plans, and retries continue to generate charges, but those costs are usually hidden by aggregate token counts.
The next stage of AI Agent evaluation should optimize total cost per successful task rather than producing another model success-rate leaderboard. That approach forces teams to improve models, prompts, tool design, and retry strategies together. It differs from traditional code benchmarks, which reward a one-time delivery. What Agent systems genuinely need is stable, inexpensive, repeatable delivery.
The 64% figure should not be treated as a universal budget benchmark. Task difficulty, model pricing, tool fees, and timeout policies all affect the ratio. It is also unclear whether the runs covered enough repository types and production traffic. The safer conclusion is to break down costs by failure cause before deciding what to optimize, rather than replacing models in response to the headline number.