Frameworks

HybridAgent Moves LLM Reasoning Into Latent Space

Together Computer has open-sourced HybridAgent, an LLM agent that performs chain-of-thought reasoning in vector space before expressing its decisions as tool calls. The release includes code, model weights, and enough training detail to evaluate its practical value and reproduce its results.

Font size

A paper surfaced on the Hacker News front page yesterday with a deceptively simple title: Hybrid Reasoning Agents: LLM agents that think in latent space, act through text. The Together Computer team has released the code and model weights. After reviewing both the paper and repository, here is what the work actually delivers.

What Is a Hybrid Reasoning Agent?

In one sentence: the reasoning stage runs in latent space as continuous vectors, while the execution stage operates in text space through readable tool calls.

Traditional ReAct and AutoGPT-style agents work like this:

Thought: 我需要先查一下北京天气
Action: search("北京天气")
Observation: 25 度,晴
Thought: 用户问的是上海,我搞错了
Action: search("上海天气")

Every step must expose its internal reasoning as a string. This approach has several familiar drawbacks:

  • It consumes tokens rapidly, with long-running tasks often spending several thousand tokens.
  • It exposes the reasoning process to users, making it more vulnerable to prompt injection.
  • Chain-of-thought is often treated as an audit trail for explainability, even though a model's verbalized reasoning does not necessarily match its internal computation.

HybridAgent takes a different approach: the model thinks in embedding space, then translates the result into readable actions. From the outside, it still issues ordinary tool calls such as search(...). The intermediate reasoning loops, however, run as continuous vectors without consuming text tokens.

The arXiv identifier is 2610.18861. The GitHub repository is at github.com/togethercomputer/HybridAgent, and the model weights are available in the togethercomputer/HybridAgent collection on Hugging Face.

What Changed in the Paper

The system uses a two-layer architecture, described clearly in Section 3 of the paper.

1. Latent Reasoning Module

During training, the architecture inserts a lightweight adapter after each layer of the base model. It takes the preceding layer's hidden state, produces a latent reasoning vector, and passes that vector to the next layer. The adapter is trained with RL; the paper compares PPO with the team's hybrid objective, which produced more stable results. Its objective is to make the generated latent vectors useful for downstream action selection.

The adapter is small. The paper reports a parameter count equal to roughly 1–3% of the base model, making it lightweight enough to swap in and out much like LoRA.

2. Action Decoder

This component translates latent vectors back into text. It uses the same underlying LLM, but switches between two output modes: standard autoregressive generation for tool calls during an action step, and latent output during a thought step. At inference time, the model switches to vector mode when it encounters the special token <|latent_step|> and returns to text mode at <|action_step|>.

The benefit is that one base model can support both textual and latent chain-of-thought without interference. In the paper's ablation study, a purely latent approach used 40–60% fewer tokens than textual chain-of-thought on long-horizon tasks while maintaining the same single-step accuracy.

Where to Find the Code and Models

The repository has a conventional structure and can be started in five minutes:

git clone https://github.com/togethercomputer/HybridAgent.git
cd HybridAgent
pip install -r requirements.txt

The inference entry point is examples/run_hybrid_agent.py. Its core usage looks like this:

from hybrid_agent import HybridAgent, HybridConfig

config = HybridConfig(
    base_model="togethercomputer/HybridAgent-7B-base",
    latent_adapter="togethercomputer/HybridAgent-7B-latent",
    action_decoder="togethercomputer/HybridAgent-7B-action",
    max_latent_steps=8,
)

agent = HybridAgent(config)

result = agent.run(
    task="查找北京今天 PM2.5 数值,并给出口罩建议",
    tools=[search_tool, calculator_tool],
)
print(result.final_answer)

Hugging Face checkpoints follow a {base}-{variant} naming scheme. The available variants are latent for latent reasoning only, action for action output only, and hybrid for the complete system. The hybrid version is the appropriate default.

For local inference, the project recommends using vLLM. The repository includes a ready-made startup script named serve_vllm.sh.

How Do the Benchmarks Look?

Section 5 compares explicit chain-of-thought with latent hybrid reasoning. Every experiment uses the same LLaMA-3-70B base model.

BenchmarkExplicit CoTHybrid (latent)Token Savings
HotpotQA (multi-hop QA)52.3%54.1%~45%
ALFWorld (text env agent)68.0%71.4%~52%
WebShop (web agent)41.7%43.2%~38%
MATH (pure reasoning)78.1%77.5%~20%
HumanEval84.0%83.7%~15%

Three findings stand out:

  • Agent benchmarks improve noticeably across the first three tasks. Latent reasoning supports repeated internal deliberation before an action, which naturally suits planning-oriented tasks.
  • Pure reasoning and code benchmarks are nearly tied. Latent reasoning offers no advantage on MATH or HumanEval, suggesting that its benefits are concentrated in workflows that require actions and iterative adjustment.
  • Token costs fall substantially. Even when accuracy remains unchanged, latent reasoning saves 15–20% on some tasks, with much larger reductions on long-horizon workflows.

Why This Work Matters

Beyond the technical novelty, the work has three practical implications.

1. Inference Cost

It changes the cost profile of long-horizon agent tasks. A 20-step ALFWorld episode using explicit chain-of-thought may consume 8K–12K tokens, compared with 4K–6K through the latent path. For production agent products, that translates directly into better gross margins.

2. A Weaker Assumption About Explainability

Much of the research literature treats LLM chain-of-thought as evidence for auditing the model's reasoning: what it says is assumed to reflect what it thinks. Latent reasoning breaks that equation. Actual internal computation is not the same as verbalized reasoning. Teams working on agent safety and compliance will need to treat chain-of-thought as potentially untrustworthy, which challenges a foundational assumption in how agents are evaluated.

3. A New Architectural Choice

Agent developers will increasingly need to make decisions at three levels: the prompt layer for instructions, the scaffold layer for state machines such as LangGraph or AutoGen, and the thinking layer, where explicit text and latent vectors become competing approaches. Together's work turns the third dimension into a practical option rather than forcing a choice between explicit chain-of-thought and RL-driven hidden-state reasoning.

Editorial Take

Latent-space reasoning is not a new idea. Researchers including Geirhos explored internal computation in transformer hidden states two years earlier, while DeepMind's Chain-of-Thought in Latent Space (2024) pursued a related direction. The real significance of Together's work is not theoretical novelty but reproducibility: open-source code, open weights, and a complete training pipeline turn the research direction into an engineering option that can be tested immediately.

A contrarian view is that the main bottleneck in today's agent frameworks is not the reasoning format but tool quality and error recovery. HybridAgent does not directly address failures in tool selection, rollback after errors, or malformed data returned by tools. Until those problems are solved, an agent may still break on Step 5 regardless of how many tokens its reasoning process saves. The work's immediate value is therefore cost reduction rather than quality improvement.

One uncertainty remains: the gap between the published scores of 54.1% and 71.4% and the results produced by the togethercomputer/HybridAgent checkpoints on Hugging Face has not yet been tested through a full benchmark run. The repository provides evaluation scripts for reproduction, but the hardware requirements are substantial: the 7B model needs at least 24 GB of GPU memory, while the 70B version requires an H100.

Sources: arXiv 2610.18861, GitHub, and HF Collections.

Source: https://arxiv.org/abs/2610.18861