Frameworks

Intelligence Index v5: A Practical Guide to Model Selection

Artificial Analysis has updated its Intelligence Index to compare model intelligence, pricing, and speed. For enterprises, the rankings are only a starting point: real selection also requires workload-specific testing, Agent trajectory analysis, and cost calculations based on actual usage.

Font size

Model leaderboards are among the most frequently misinterpreted tools in AI.

This guide breaks down the evaluation dimensions, cost model, and Agent trajectory analysis in Intelligence Index v5, giving technical leaders a practical model-selection process.

Intelligence Index v5: A Practical Guide to Model Selection

Artificial Analysis recently released Intelligence Index v5. It is not a new model, but the fifth revision of its benchmark and ranking framework for comparing the capabilities and cost of different large language models.

When the update reached the front page of Hacker News, it received 54 points and 61 comments. Developers were interested in more than who ranked first. They wanted to know whether the rankings remained comparable across models and whether they could support practical selection decisions.

One limitation should be noted upfront: the available news summary does not include v5's model rankings, metric weights, or score changes. What can be confirmed is the framework update and its three evaluation dimensions: reasoning quality, output price, and response speed. The exact weights and model positions still need to be verified against the original article.

v5 Is Not Just Another Results Table

Intelligence Index v5 is a framework upgrade, not a model release.

If it simply replaced the test set, reran the models, and published a new leaderboard, it would offer little more than a time-limited report card. A framework upgrade instead changes how models are compared.

Traditional model rankings often compress capability into a single integer. That is convenient, but it loses substantial information.

One model may excel at reasoning tasks while remaining too expensive for high-frequency calls. Another may have a slightly lower overall score but respond faster, making it more reliable when integrated into a customer-support Agent. If a leaderboard provides only final rankings, procurement teams must still break down the underlying data themselves.

v5 at least moves the comparison beyond a single score by presenting capability and cost side by side. The Artificial Analysis summary explicitly states that this version is designed to compare the capabilities and usage costs of different large language models.

Original source: Artificial Analysis's Intelligence Index v5 article.

The 54 points and 61 comments on HN provide additional evidence that developers are engaging with the leaderboard. Their discussion, however, centers on comparability.

Overall Scores Can Hide Procurement Costs

For enterprise model selection, at least three questions need to be separated:

  1. How strong is the model's reasoning?
  2. What does output cost?
  3. How fast does it respond?

Combining these factors into one score makes rankings look simple, but it does not necessarily make procurement decisions simpler.

Suppose two models differ by only a few points, but one costs 80% less and the other has half the time to first token. For an AI Agent handling millions of calls per day, that difference may matter more than those few points of benchmark performance.

The trade-off between quality and price is not fixed. High-frequency, well-defined tasks with costly failures can have a different optimal cost curve from one-off research tasks.

A low-scoring but inexpensive model is not necessarily a low-value model. It may be unsuitable for difficult benchmark problems while remaining effective for repetitive work such as classification, extraction, rewriting, and routing. Within an Agent's main loop, many requests do not involve solving an intellectual challenge. The model functions more as an imperfect but callable executor.

When reviewing Intelligence Index v5, the first priority is therefore not to look at the top-ranked model. It is to determine what each dimension means within the enterprise's own calling patterns.

Consider a customer-support Agent that handles 200,000 conversations per day, with each conversation requiring 2,000 input tokens and 300 output tokens. Even when two models have similar benchmark results, multiplying output price by call volume can turn the difference into a direct impact on the quarterly budget.

If the current news summary does not provide each model's actual price, ranking alone cannot establish which option is cheaper. The benchmark framework offers a consistent starting point, but vendor invoices and actual token usage must still be used for final validation.

AI Agents Must Be Evaluated by Trajectory

This is the most overlooked layer of v5, and potentially the most important for enterprises.

A static benchmark measures a single response. An AI Agent performs a sequence of actions: understanding a goal, calling tools, interpreting results, identifying deviations, and deciding whether to retry or stop.

Choosing the wrong tool on the first attempt does not immediately end the task. The system may enter a retry loop, consuming more tokens and waiting longer before ultimately delivering an incorrect result.

Several engineering questions can stand between a leaderboard score and real-world task completion rates:

  • Are the tool descriptions clear?
  • Is the parameter format stable?
  • Does the context contain irrelevant information?
  • After a failure, should the system retry, switch models, or escalate to a person?
  • Can timeouts and error states be captured reliably?

Switching to a higher-scoring model does not automatically solve these issues.

A common project pattern involves a top-ranked model with an 82% completion rate and a second-ranked model with an 88% rate. The second model is selected for production because the leader produces less consistent tool-call formats, resulting in more parsing failures and human escalations.

The relevant metric is the trajectory, not only the final answer.

At minimum, teams should track four metrics:

  • Task completion rate
  • Average number of tool calls
  • Failure and retry rate
  • End-to-end cost per successful task

The last metric has the greatest procurement value because it places token consumption, latency, infrastructure expenses, and human-escalation costs on the same basis.

A cheaper model that requires four retries per task may cost more than a model priced three times higher if the latter succeeds on the first attempt.

Treat the Index as a Filter, Not a Verdict

Intelligence Index v5 is not the final answer to enterprise model selection. Its more useful role is as a candidate filter.

Step 1: Build an Internal Task Set

Do not copy a public benchmark directly. Anonymize real tasks and turn them into several hundred repeatable samples, such as support tickets, code changes, data extraction, report generation, or routing and tool calls within LangGraph.

Every sample should have explicit acceptance criteria. Record intermediate failures rather than evaluating only the final output.

Step 2: Use v5 to Narrow the Field

First exclude models that clearly violate quality, speed, or budget constraints. Then retain a small group of candidates. A shared framework makes this stage more controlled than allowing individual teams to test models based on personal preference.

Step 3: Run Realistic Load Tests

Test at least three workload profiles:

  • P50 and P95 latency under peak concurrency
  • Sustained throughput and token consumption
  • Tool-call failures, retries, and task completion

Step 4: Recalculate Costs from Actual Billing

Price evaluation should not rely solely on vendor list prices. Calculate costs using the input, output, cached, and retried tokens generated by internal workloads. Rate limits, volume discounts, regional deployment, and context-caching strategies must also be included.

A simple, reproducible screening script can begin by accepting public score data in a consistent format:

from dataclasses import dataclass

@dataclass
class ModelCost:
    name: str
    quality_score: float
    output_price_per_million: float
    tokens_per_second: float

def shortlist(
    models: list[ModelCost],
    min_quality: float,
    max_price: float,
    min_speed: float,
) -> list[ModelCost]:
    return [
        model
        for model in models
        if model.quality_score >= min_quality
        and model.output_price_per_million <= max_price
        and model.tokens_per_second >= min_speed
    ]

models = [
    ModelCost("model-a", 78.0, 1.50, 95.0),
    ModelCost("model-b", 81.0, 4.00, 70.0),
]

result = shortlist(
    models,
    min_quality=75.0,
    max_price=2.0,
    min_speed=80.0,
)

for model in result:
    print(model.name)

This code performs threshold filtering rather than a complete evaluation. It deliberately avoids collapsing quality, price, and speed into a single score. Enterprises should define constraints first, then test the remaining candidates against internal workloads.

Editorial Perspective

The main value of Intelligence Index v5 may not be a reshuffled leaderboard. Its value is giving procurement and engineering teams a shared language for discussing quality, price, and speed without hiding them behind an opaque total. What distinguishes it from a conventional leaderboard is not a new test set, but the beginning of cost comparisons alongside capability comparisons.

The conclusion should remain cautious: the more unified a framework becomes, the easier it is to forget that uniformity itself is a simplification. The definition of v5's weights, the capabilities covered by its test set, and the timing of model-version freezes can all affect rankings. Because the current summary does not include specific weights or score changes, any claim that a particular model has overtaken another should wait until the original data is available.

For AI Agents, the next step is not merely to continue tracking public leaderboards. It is to connect failure trajectories, real latency, and billing data in a single evaluation. Enterprises are not buying a higher benchmark score; they are buying a more reliable task completion rate and a lower cost per successful task.

Source: https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v5