Frameworks

17,000 Runs Later: What Tools Do Claude, Codex, and Cursor Actually Reach For?

Armature Tech ran 17,000 tasks across Claude, Codex, and Cursor and logged which tools each agent installed and called on its own. The results reveal some counterintuitive patterns — including the finding that pricier models tend to be the laziest tool users.

Font size

Most agent benchmarks answer a familiar question: which model writes better code? A recent post from Armature Tech asks a different one — what tools do coding agents choose for themselves when no one is watching?

Across 17,000 runs covering real engineering work (reading repos, writing tests, fixing dependencies, patching bugs), they logged every npm install, CLI invocation, and MCP server the agents touched. The result is a map of tool preferences — measured from actual logs, not marketing pages.

The Setup: Watching Agents Shop, Not Watching Demos

Standard evaluation works like this: hand the agent a prompt, check if the output is correct. HumanEval, MBPP, SWE-bench — all follow that pattern.

Armature took a different angle. Instead of grading output quality, they recorded the tools each agent proactively installed and called. The agents decided which packages to pull in, which CLIs to invoke, and which MCP servers to connect to. After each task, the tool-call logs were aggregated and counted.

The advantage of this approach is that it cuts through vendor messaging. The feature lists Anthropic, OpenAI, and Cursor publish are one thing; the tools their agents actually reach for across 17,000 tasks are another. The two often don't match.

Tool Preferences Vary More Than Expected

Armature grouped high-frequency tools into categories. A few highlights:

The shared basics

  • File operations: Read, Write, Edit, Glob, Grep — universal.
  • Git toolchain: git status, git diff, git commit — used across the board.
  • Test runners: pytest for Python tasks, npm test for JS tasks.
  • Package managers: pip, npm, pnpm depending on language, but all called frequently.

None of this is surprising. The interesting part is where the three diverge.

Where they diverge

  • Claude leans heavily on LSP-related tools. It tends to read the code first — jumping to definitions, inspecting types, chasing references — before touching anything. Static-check calls like tree-sitter, mypy, and tsc --noEmit appear at noticeably higher rates than with the other two.
  • Codex prefers to edit directly. Its tool-call chains are shorter, linting is rare, and commits tend to come right after edits. It looks like the "verify later" step has been removed.
  • Cursor calls IDE-integrated tools most often — tab jumps, multi-file diffs, in-editor test runs. It depends heavily on its own editor context.

Quirks unique to each

  • Claude often installs ruff and black on its own, even when the task doesn't request them.
  • Codex tends to batch multiple changes into a single commit, producing the lowest tool-call count of the three.
  • Cursor repeatedly calls "read currently selected file" in long tasks — it's deeply tied to IDE state.

These aren't vendor claims. They come from mining 17,000 logs.

A Counterintuitive Finding: Pricier Agents Are Lazier

The most surprising result involves model capability and tool usage.

The naive expectation is that Claude Opus 4, being the most capable, should install the most tools and use the widest range. The data says otherwise.

Armature's numbers show that more capable models make fewer tool calls.

  • Claude — especially at Sonnet and Opus tiers — completes the same tasks with fewer tool invocations. It prefers to gather all needed context in one call rather than ping-pong.
  • Codex's tool-call count is sometimes 1.5–2× Claude's on equivalent tasks.
  • Cursor sits in the middle, but its IDE-tool share is disproportionately high.

This runs counter to the intuition that stronger models should be busier. A more accurate reading is that stronger models are more precise about tool selection. They know how to get everything they need in one shot; weaker models need more rounds to guess their way forward.

In other words, tool-call count is not a proxy for effort. Lower counts can signal higher efficiency.

A related pattern: more capable (and more expensive) models install new dependencies less often. They prefer to reuse what's already in the project rather than running npm install on a fresh pile of packages. That's good news for production — dependency bloat is a hidden cost in agent systems.

What This Means for Builders: Benchmarks Aren't Enough

For teams shipping agent products or wiring agents into CI, the data has a few direct implications.

1. Estimate token cost from tool-call patterns, not model price.

Model pricing is quoted in input/output tokens. But for agents, the bulk of token spend is tool-returned content — file bodies, grep results, test output. Claude calls fewer tools, but each call may pull more content. Total cost depends on task shape.

# A rough cost-estimation framework
def estimate_cost(agent_runs):
    total = 0
    for run in agent_runs:
        # Average tokens per tool call
        avg_tool_tokens = run['total_tokens'] / max(run['tool_calls'], 1)
        # Model price
        model_cost = run['model_price_per_1k'] * run['total_tokens'] / 1000
        total += model_cost
    return total

Numbers from this kind of estimate will diverge from raw "which model is more expensive" comparisons.

2. Don't apply tool whitelists uniformly.

Armature's data suggests splitting tool sets by task profile:

  • Read-heavy, edit-light tasks → Claude-style toolset (strong LSP, strong static checks)
  • Rapid iteration tasks → Codex-style (short call chains, frequent commits)
  • Deep IDE integration scenarios → Cursor's path

3. Watch out for tool-pool bloat.

More tools available doesn't mean better outcomes. The data pushes back: Claude stays more stable in lean tool environments — it knows what to touch and what to leave alone. When the tool pool expands, all three agents' reliability drops.

Practical rule of thumb: start with 5–10 core tools at launch, add more based on observed task performance. Don't expose the entire ecosystem upfront.

Why This Matters: Agent Evaluation Is Shifting

This deserves a separate note because the focus of agent evaluation is quietly moving.

For the past two years, the dominant question has been "whose code passes more tests." HumanEval, MBPP, SWE-bench — leaderboards ruled. But those benchmarks treat agents like answer-writing machines and ignore a basic fact: agents work inside tool ecosystems, and how they pick tools directly affects production cost and stability.

Armature's method offers a different ruler. Instead of counting correct answers, it counts what the agent reached for across 17,000 real tasks. That ruler sits closer to how these systems actually get used.

The likely next move: agent evaluation pivots from output quality to tool-use efficiency. Concrete metrics might include tokens per task, tool-call redundancy rate, dependency-bloat index. Some teams are already working on this.

Caveats on Armature's approach: the 17,000 tasks vary in type, codebase, and agent configuration. Claude's "fewer tools" preference might become a liability on very long tasks where the context it pulls in one shot bumps against token limits. The data is cross-sectional, not causal.

There's also an open question: are these preferences baked into model architecture, or shaped by training data? Would a different fine-tune produce different tool habits? Armature doesn't answer this, and neither can I.

But one thing holds: watching an agent's tool-call logs is more reliable than reading vendor marketing.

Source: https://armature.tech/blog/which-tools-coding-agents-install