Opinion

How to Choose an AI Agent Model by Cost per Task

Village.dev argues that closed models now lead on price-performance. For Agent teams, however, the relevant metric is the total cost of each successfully completed task, including retries, tool calls, and human intervention.

Font size

Closed models are taking over the price-performance frontier. For AI Agent teams, leaderboard position matters less than how much it costs to complete a task successfully.

Closed Models Lead on Value: Agent Teams Need a New Cost Model

Closed models are taking over the price-performance frontier.

This is not a new model launch, an official benchmark, or a peer-reviewed study. It is closer to a procurement map: place inference pricing and model capability on the same chart, then identify which option delivers reliably on real-world tasks.

For AI Agent teams, a lower unit price is only the surface. The factor that ultimately determines gross margin is the cost of each successfully completed task.

What Changed on the Price-Performance Frontier

A Village.dev article makes the case that closed AI models now dominate the price-performance frontier.

The comparison is not based on a single benchmark result or how intelligent a model's answers appear. The question is which model offers better value when inference price and capability are considered together.

The analysis also examines the enterprise-level shift: why closed models may replace open models as the preferred value option in enterprise AI procurement, and how that changes the deployment cost of Agents.

This differs from conventional model leaderboards.

A typical leaderboard asks: “Who scores higher on a fixed test set?”

A procurement decision asks: “Given a budget and latency requirements, who completes the most tasks?”

The two questions are related, but they are not interchangeable. A model may perform exceptionally well in a single evaluation yet cost more per completed task if its output is inconsistent, prone to retries, or requires longer context.

As a result, the most important development on the price-performance frontier is not how far closed models have moved on the chart. It is that enterprises are beginning to select models using a measure closer to unit economics.

See the original analysis on Village.dev: Closed AI Models Now Dominate the Price-Performance Frontier.

Do Not Mistake Token Price for Task Cost

The easiest cost calculation to get wrong in an Agent product is token pricing.

Providers usually list prices per million input tokens and per million output tokens. These figures are useful for budgeting and initial model screening. But an Agent does not ultimately deliver tokens; it delivers a completed task.

The unit cost of a task can be expressed approximately as:

Cost per task =
(input token cost + output token cost + tool call cost) × total attempts
+ human intervention cost
+ loss from failures and latency

At least three variables can change the result.

First, retries. If a task has an 80% success rate, it takes an average of 1.5 attempts. If the success rate falls to 60%, the average approaches two attempts. A model with a low unit price but longer inputs and less consistent outputs may consume the budget through repeated attempts.

Second, tool calls. If an Agent needs to query a database, run code, control a browser, or generate files, the model price is only one part of the total cost. Its ability to recognize when to stop, use tools correctly, and recover from failures all affect how many steps each task requires.

Third, context length. A longer context is not necessarily more cost-effective. Every turn costs more to process, and filling the context with irrelevant material may reduce the model's ability to retrieve critical information.

Fourth, completion rate. This means end-to-end completion, not benchmark performance. If a user asks the Agent to “organize this week's sales data and generate a report,” a correct answer that omits an attachment, violates the required format, or fails to handle an exception still represents a business failure.

Therefore, a closed model that is “cheaper per call” is not necessarily cheaper per task. The reverse is also true. An open model may have a lower cost per completed task if it is deployed reliably and benefits from high cache hit rates.

Are Open Models Still Worth Deploying?

This decision should not be reduced to open versus closed.

For high-frequency, online Agents, the price advantage of closed models is usually more direct. Development teams avoid purchasing GPUs, scaling inference infrastructure, and handling operations. If the API's cost per task is low enough and the provider's reliability and data terms are acceptable, a closed API is often easier to launch.

Open models remain worth deploying or retaining in several situations:

  • Data cannot leave the controlled environment. Healthcare, finance, government, or internal enterprise code requires clear boundaries for data residency, deletion, and access. The value of local deployment includes control, not just price.
  • High volume and stable traffic. Once call volume is sufficient to amortize GPU costs, the fixed cost of self-hosting becomes manageable.
  • Strict latency requirements. Local inference can avoid public network variability for fixed workloads, short contexts, and high-frequency requests.
  • Effective caching is possible. Repeated questions, fixed retrieval results, and deterministic prefixes can reduce the marginal cost of a self-hosted model.
  • Custom weights are required. Open weights offer greater flexibility when a business depends on a specific language, behavior, or controlled generation.

Local deployment also comes with hidden expenses: GPUs, quantization losses, monitoring, upgrades, capacity planning, and disaster recovery.

This is better treated as a portfolio decision than a loyalty test:

  • Use closed APIs for general, complex, low-frequency tasks.
  • Use self-hosted models for high-frequency, stable, cacheable workloads.
  • Route sensitive data according to compliance requirements.
  • Keep human approval for high-risk actions.

Such an architecture does not rely on a single model. It also reduces exposure to provider price increases, rate limits, or API changes.

What the HN Debate Revealed

The Village.dev article received 47 points and 34 comments on Hacker News. The discussion focused on two issues: whether the price-performance advantage of closed models is real and whether the underlying evaluation method is reliable.

HN page: item?id=50387546.

The points of contention were predictable.

First, this was an analysis based on model pricing and performance data, not a model launch, official benchmark, or peer-reviewed study. Its sample selection, task weighting, and normalization of API prices all require further verification.

Second, public evaluations often use tasks with relatively well-defined distributions. Enterprise Agents combine web navigation, data cleaning, code modification, customer-support decisions, and document generation. A model's strength in one category does not mean it will lead on real enterprise traffic.

Third, a price-performance curve is time-sensitive. Input prices, output prices, and context windows can all change within a week. Drawing long-term procurement conclusions directly from a static table carries considerable risk.

The most useful outcome of the HN debate was not declaring a winner between open and closed models. It forced evaluators to answer several basic questions: Which models were tested? How were they tested? How were failures billed? Does production traffic have the same distribution as the test set?

Those questions matter more than platform slogans.

A Procurement Checklist for Agent Teams

Do not select a model directly from a public leaderboard. Start by running an end-to-end evaluation with the team's own task set.

At minimum, record the following metrics:

DimensionWhat to recordWhy it matters
Cost per successful taskThe full-chain cost of each successful completionDirectly affects unit economics
P95 latencyThe long-tail latency from trigger to task completionInfluences user experience and concurrency capacity
Retry rateAverage attempts and recovery success rateMeasures reliability and hidden model costs
Tool callsThe number of calls and failed calls needed to complete a taskShows whether the Agent is making progress
End-to-end completion rateThe percentage of tasks accepted by the businessPrevents reliance on model scores alone
PortabilityThe amount of code and prompt rework required after switching modelsReduces provider lock-in risk
Data and complianceRetention, training use, regional, and deletion policiesDetermines whether the model can enter production

For an evaluation, assign the same set of tasks randomly to candidate models and standardize the maximum step count, tool permissions, and timeout settings. Failures must not be silently excluded, and the evaluation should never report only the average token usage of successful samples.

For more granular attribution, a minimal script can track calls and retries:

from dataclasses import dataclass

@dataclass
class RunStats:
    attempts: int = 0
    input_tokens: int = 0
    output_tokens: int = 0
    tool_calls: int = 0
    tool_failures: int = 0
    completed: bool = False

    def attempt(self) -> None:
        self.attempts += 1

    def usage(self, input_tokens: int, output_tokens: int) -> None:
        self.input_tokens += input_tokens
        self.output_tokens += output_tokens

    def tool_result(self, failed: bool = False) -> None:
        self.tool_calls += 1
        if failed:
            self.tool_failures += 1

    def finish(self, completed: bool) -> None:
        self.completed = completed

This code does not call the model. It simply prevents inconsistencies in the evaluation methodology from being lost. The results can then be calculated as follows:

successful_costs = [
    cost
    for cost, stats in results
    if stats.completed
]

avg_cost_per_success = (
    sum(successful_costs) / len(successful_costs)
    if successful_costs else float("inf")
)

A more reliable approach also includes failed tasks in the cost denominator. Otherwise, a model that only completes simple tasks can appear unusually inexpensive.

Editorial Perspective

Closed models taking the price-performance frontier reflects a change in how Agents are procured, not a sudden decline in open-model capability. Agents translate model differences directly into gross margin: saving a fraction of a cent on one call is less valuable than avoiding two retries on one task.

The distinguishing feature of this shift is the change in the evaluation unit from “model capability” to “successful delivery.” Model parameters, public leaderboards, and token prices remain relevant, but procurement teams will increasingly ask: How much does each successful task cost? Who absorbs the cost of failures and retries?

The mainstream architecture of the future is unlikely to commit to a single provider for long. Closed models may handle most general traffic, while open models retain roles in sensitive data, high-frequency caching, and customization. Whether Village.dev's methodology is sufficient to prove that closed models lead across the board remains uncertain. The article lacks an official benchmark and peer review, and its public tasks may not match the distribution of enterprise workloads. Credible conclusions will emerge only when teams reproduce the results with their own data.

Source: https://village.dev/blog/closed-ai-models-now-dominate-the-price-performance-frontier/

Source: https://village.dev/blog/closed-ai-models-now-dominate-the-price-performance-frontier/