Tools

ThinkingCap Cuts Gemma Reasoning by 69%—Does It Save?

ThinkingCap reduced reasoning effort by 39%–69% on Gemma-4-26B-A4B without lowering accuracy. Whether those gains translate into lower production costs depends on the full task trajectory.

Font size

ThinkingCap enables Gemma-4-26B-A4B to reason 39%–69% less while maintaining the same accuracy. But a benchmark reduction does not mean production bills will fall by the same amount. This article breaks down the economics of reasoning compression for Agents.

Reasoning compression sounds straightforward: generate fewer thinking tokens without reducing accuracy.

ThinkingCap reported exactly that result with Gemma-4-26B-A4B: reasoning volume fell by 39%–69%, with no change in accuracy. The model was not swapped, and it was not asked to fob users off with shorter answers. The change happened during inference.

For an ordinary chat product, this might mean slightly faster generation. For an AI Agent, it could matter much more. Agents often go through multiple rounds of decisions, tool calls, and result processing. Every additional thinking token in each round may be multiplied across calls, then compounded by latency and cost.

However, 69% is an experimental result, not a discount that can be applied directly in production.

First, the experimental results: no model swap

The tested model was Gemma-4-26B-A4B. It has 26B total parameters, with A4B active parameters. A4B can be understood as activating roughly 4B parameters for each inference rather than computing across all 26B.

ThinkingCap's headline result was a 39%–69% reduction in reasoning volume with no accuracy loss.

These three numbers need to be considered together.

If accuracy fell by 10%, saving 40% of inference would usually not count as a success. Agent task distributions are uneven, and a small number of incorrect decisions can trigger the wrong tool calls, repeated actions, or even prevent a workflow from completing. ThinkingCap is trying to demonstrate a different approach: determine how much reasoning budget the current problem needs, then allocate it dynamically.

This is not a swap of Gemma-4-26B-A4B for a smaller model. The model, tasks, and accuracy baseline remain unchanged; what is compressed is the thinking process in between.

The original experiment was published on bottlecapai.com: Gemma-4-26B-A4B with 39-69% less thinking at same accuracy.

A related link was posted on bottleyai.com. Its Hacker News page currently shows 69 points and eight comments.

The discussion is still small, so the 39%–69% result is better treated as a technical signal awaiting replication than as a broadly validated industry conclusion. Key questions remain: what reasoning budget controller was used, whether it compressed reasoning tokens or entire blocks of thought text, and which tasks were included in the accuracy calculation.

Why “thinking less” may be worth more than “thinking differently”

There are two common ways to change how much a model thinks.

The first is to reduce the number of reasoning rounds. For example, an Agent can produce an answer and then decide whether to continue. This is cheaper, but if the first decision is wrong, the rest of the chain may go off course.

The second is to shorten each reasoning pass. Inference becomes more compact, but complex tasks may fail because too little information was considered.

Dynamic reasoning attempts to avoid an all-or-nothing switch: think less about simple questions and more about difficult ones.

This is especially important for Agents. A typical workflow might look like this:

User request
  → Read context
  → Select tool
  → Call API
  → Parse result
  → Determine whether the task is complete
  → Retry or correct if needed

A 10% reduction in inference within one round does not translate cleanly into a 10% reduction across a six-step tool chain. It may affect time to first token, decision time before each tool call, and the probability that the entire chain times out.

Failures are often where the cost lies. After making the wrong decision, an Agent may take two extra steps: reinterpreting the tool parameters, calling again, and then waiting for a result. The most expensive part of a production bill is often not the normal path, but these additional turns.

The real benefit of reasoning compression may therefore be fewer unproductive thoughts and detours, rather than simply fewer tokens.

That still needs to be measured separately. A reduction in average reasoning volume does not automatically imply lower average completion time, nor fewer error-recovery attempts.

Calculating the cost in Agent workloads

I would break the calculation into at least four parts.

1. Token costs

Tokens are the easiest component to calculate, but there are three types to distinguish: input, output, and cache hits.

Lower reasoning volume primarily affects thinking tokens generated by the model. If a tool returns a long document, the Agent will place that result back into the context. Input tokens may not decrease and could even rise if the Agent takes an extra round.

A production bill cannot simply state:

Original cost = total tokens × unit price

At a minimum, it should be calculated as:

Cost per task =
  input tokens for each round × input unit price
  + output tokens for each round (including thinking) × output unit price
  + tool-call fees
  + retry and sandbox fees

2. Number of calls

The number of rounds determines whether a task is genuinely cheaper.

Suppose ThinkingCap generates 50% fewer thinking tokens, but a benchmark task makes one additional tool call on average. That extra call could erase the token savings. It could also increase API rate limiting, retries, and state-synchronization overhead.

Evaluation therefore cannot stop at “fewer tokens per response.” It must measure the complete trajectory from the start of a task to successful acceptance.

3. Context replay

Many Agent frameworks resend earlier turns, tool outputs, and system prompts. Shorter reasoning may slow context growth, but one additional round brings the replay cost back.

The calculation changes again if the system uses a prompt cache. A cache hit is inexpensive, but it still involves waiting. Recomputing after a cache miss may not be more economical than using a shorter context in the first place.

4. Timeouts and reliability

Latency is usually more closely tied to user experience. Reasoning time within a tool chain accumulates serially, so even if the final output arrives sooner, the overall process may still exceed the caller's timeout.

Once a timeout occurs, the Agent may trigger a retry. Retries after timeouts often cost more than a normal execution. More importantly, do timeouts affect only the average? Do long-tail tasks become less stable?

I would track p50, p95, and p99 latency rather than looking at the average alone. Completion rate, retry rate, and tool-call count should also be recorded.

A more credible production validation uses a fixed task set and compares complete trajectories:

MetricThinkingCapBaseline
AccuracyNo lower than baselineBaseline
Total input tokensMeasuredMeasured
Total output and thinking tokensMeasuredMeasured
Average tool callsMeasuredMeasured
p50 / p95 completion timeMeasuredMeasured
Retry and timeout ratesMeasuredMeasured
Total cost per successful taskMeasuredMeasured

The final row matters more than the price per token. Agents should be evaluated on the cost of successfully completed tasks.

The limits of the approach: beyond accuracy

Unchanged average accuracy shows only that overall results on the experimental set did not materially deteriorate. It cannot replace evaluation by task, difficulty, and risk level.

Complex reasoning

Mathematical proofs, code repair, and multi-constraint planning often require longer reasoning chains. If a budget controller compresses too aggressively, it may turn “not figured out yet” into a confident wrong answer.

Simple and difficult tasks need to be compared separately. Token reductions on simple question-and-answer tasks should not conceal lower success rates on complex work.

Long-running tasks

A 69% reduction in thinking on a short task may be valuable. For an Agent that runs for 15 minutes and makes dozens of tool calls, each local saving is small. What determines cost is whether the overall task trajectory becomes shorter.

Failure recovery

The most valuable Agents are not those that never fail, but those that recover from failure.

If ThinkingCap compresses the model's reflection on error logs, the Agent may repeat the same mistake faster. Testing should measure more than the final success rate. It should also examine:

  • Whether the same error recurs
  • Whether retries converge
  • Whether the Agent chooses an alternative tool
  • Whether it asks the user to clarify critical ambiguities

Tool-call safety

Thinking less is not always better. When an action involves deleting data, making a payment, sending a message, or changing a production configuration, one additional validation step may be worth more than a few extra tokens.

Tools can be divided into low- and high-risk categories. Low-risk retrieval and query tasks can benefit from aggressive compression. High-risk write, payment, and deletion actions should retain longer reasoning or use an independent verifier. A single budget policy should not cover every action.

Fewer compressed reasoning tokens may also affect the model's ability to produce a complete intermediate trace. Log interpretability and post hoc auditing need separate evaluation. An unchanged final answer is not enough to dismiss changes in process quality.

Editorial perspective

ThinkingCap is worth watching not only because of the 69% figure, but because it reframes the question from “can the model become smarter?” to “can the system allocate reasoning resources according to task difficulty?” This resembles online scheduling: return simple requests quickly, add budget for complex ones, and route high-risk actions into verification branches.

It also differs from earlier quantization approaches. Quantization generally trades model capability for lower memory and compute costs before deployment. The savings are stable, but so are the capability boundaries. Shortening context primarily reduces input overhead. Dynamic thinking budgets instead govern runtime decisions, which offers more flexibility in theory but is harder to evaluate.

My view is that reasoning compression will gradually become part of Agent infrastructure, but it will not succeed through a single universal percentage. A useful controller should adjust dynamically based on tool risk, remaining task time, and failure probability rather than optimizing average token reduction alone.

It remains unclear whether ThinkingCap's gains will hold across models and tasks. The 69 points and eight comments also indicate that it is still in an early validation stage. If production bills do not fall in tandem, or if p95 latency and recovery performance deteriorate, then “thinking less” is merely writing fewer tokens—not producing a cheaper Agent.

Source

Source: https://bottleyai.com/post/thinkingcap-gemma-4-26b/