LLMs

Mercury 2.5 Hits 1000+ tok/s with a Diffusion-Based LLM

Inception Labs' diffusion-based Mercury 2.5 generates code at over 1000 tokens per second, breaking sharply from the autoregressive approach used by GPT and Claude. Worth a closer look at what this architectural split actually means.

Font size

Inception Labs released Mercury 2.5, a diffusion-based architecture pushing 1000+ tok/s, purpose-built for code generation. That number is essentially unheard of among mainstream autoregressive models. After digging through the official blog and the Hacker News thread, here is what stands out.

Where Mercury 2.5's Speed Actually Comes From

In official benchmarks, Mercury 2.5 hits 1000+ tokens/second on code completion tasks. For comparison:

  • GPT-4o: ~80-100 tok/s
  • Claude 3.5 Sonnet: ~60-80 tok/s
  • Mercury 2.5: 1000+ tok/s

That is a 10-15x gap, and it does not come from incremental optimization. It comes from architecture.

Autoregressive models (GPT, Claude, Gemini all run this way) emit tokens one at a time, computing the next only after the previous lands. It is fundamentally serial. Diffusion-based LLMs flip the script: they generate a rough draft first, then iteratively denoise and refine, like a painter laying down base colors before adding detail. The mechanism is inherently parallel, since multiple tokens can be refined simultaneously.

Code is a particularly good fit for this parallelism. Code blocks carry obvious structural cues (indentation, bracket matching, keywords), so a diffusion model can lay out the skeleton of an entire function in one pass, then come back to fill in the details. That is a different beast from writing prose. Diffusion models will not displace novelists anytime soon, but on code completion they enjoy a genuine structural advantage.

Inception Labs' own benchmarks show Mercury 2.5 maintaining accuracy comparable to GPT-4o on HumanEval and MBPP while cutting latency by an order of magnitude (source: inceptionlabs.ai/blog/introducing-mercury-2-5).

Diffusion Is Not New, So Why Now?

Diffusion models already won image generation years ago (Stable Diffusion, Midjourney, the usual suspects). Text-based attempts date back to 2023, and Mercury 1 shipped last year without much fanfare.

Mercury 2.5 landed on the HN front page (source: news.ycombinator.com/item?id=49616354). The discussion clustered around a few themes:

  1. "Is this the RISC vs CISC moment?" Someone drew the CPU analogy: autoregressive is CISC (complex instructions, serial execution), diffusion is RISC (simplified, parallel). The metaphor is rough, but the direction is right.
  2. "What about the ecosystem?" Mainstream inference stacks like vLLM and TensorRT-LLM are written for autoregressive models. Diffusion needs adapters.
  3. "Can it be fine-tuned?" Practically no usable fine-tuning toolchain exists today.
  4. "Is the quality actually there?" Benchmarks look good, but real-world code generation is the real test.

A few HN commenters noted that Mercury 1 already worked well enough as a Cursor autocomplete backend, though it struggled with longer, more complex logic. The improvements in 2.5 on that front are worth tracking.

Code Is the Main Event

Inception Labs is not pretending otherwise. Mercury 2.5 does no general chat, no long-document summarization. It targets code completion and code generation, period.

The reason is simple: tools like Cursor, Copilot, Continue.dev, and Tabnine live or die on latency. A user typing and seeing a 300ms pause on the cursor experiences that as a broken product. Code completion is even more sensitive than search. The user's mental model is "I am typing," so any perceptible wait registers as lag.

Diffusion models stack up well in this scenario:

  • Parallel generation: tokens for an entire function can be computed together rather than streamed one-by-one
  • Structural awareness: code's syntactic constraints make the denoising process more efficient
  • Truncatable: even mid-computation outputs tend to be usable code snippets

A typical call looks roughly like this (modeled after the official docs' API style):

from inception import Mercury

client = Mercury(api_key="...")

response = client.completions.create(
    model="mercury-2-5",
    prompt="def fibonacci(n):\n    ",
    max_tokens=128,
    temperature=0.2,
)

print(response.choices[0].text)

Response times typically land in the tens of milliseconds. Inside an IDE like Cursor, the user experience approaches "appears as you type."

One caveat: Mercury 2.5 is heavily code-oriented in its training data, so it will sound painfully awkward in casual chat or email drafts. Treating it as a general-purpose LLM sets the wrong expectations.

The Commercial Problem: Fast Is Cool, But Who Pays?

This is the contrarian part worth highlighting.

Mercury 2.5's speed is genuinely impressive, but three obstacles stand between it and real production deployment:

1. Almost no ecosystem

vLLM, TGI (Text Generation Inference), TensorRT-LLM, and the rest are built for autoregressive models. Running Mercury in production means building custom inference infrastructure, which is enough to scare off most companies before they start.

2. API incompatibility

The OpenAI API is the de facto standard. LangChain, LlamaIndex, and every Agent framework default to OpenAI's protocol. Mercury's native API does not conform, so integration is not free. Inception Labs provides an OpenAI-compatible layer, but expect feature trade-offs.

3. No fine-tuning toolchain

Enterprise teams who want to fine-tune on their own codebase? There is no mature path today. LoRA, QLoRA, and friends were designed for autoregressive Transformers. Diffusion fine-tuning requires new research.

My read: much of that speed lead will be clawed back in 2026-2027 as autoregressive models adopt better caching and speculative decoding. Anthropic's prompt caching already cuts repeated-prefix costs by 10x, and speculative decoding (small-model draft + large-model verify) routinely delivers 2-3x speedups. When GPT-5, Claude 5, and successors bake these optimizations in natively, Mercury's 10x advantage shrinks to 2-3x, and at that point speed stops being decisive.

Inception Labs' real moat should be positioning as the AI-coding-specialized model, not simply the fastest one. If they can build a product that is materially better in the IDE workflow (deep Cursor integration, say), the story holds. Otherwise the speed alone will not carry it.

Bigger Picture

Mercury 2.5 matters more than a typical LLM release because it represents a genuine architectural fork, not just another model in the same lane.

For five years, LLM progress has run almost exclusively on the autoregressive track, with everyone competing on parameters, data, and RLHF within the same paradigm. Mercury (alongside earlier Mistral experiments and Google's diffusion-for-text work) demonstrates that a fundamentally different path is not just viable but offers distinct advantages: speed, structured output, parallelism.

The CPU analogy holds up reasonably well. In the short term, CISC (autoregressive) will keep dominating thanks to ecosystem maturity. RISC (diffusion) will carve out territory in specific scenarios where low latency, structured output, and parallelism matter. Long term, the two may converge. A next-generation model might use autoregression for high-level planning and diffusion for local refinement.

Practical takeaway for builders: do not put all your eggs in the Transformer-autoregressive basket. If your product is latency-critical (IDE completion, real-time voice, robotics control), the diffusion route deserves a serious evaluation. At minimum, run a few prompts through Mercury's API and see how it feels.

What remains unclear: diffusion model performance on long contexts (>32k tokens) and complex reasoning tasks is sparsely documented. Official benchmarks lean toward short tasks like HumanEval. Real-world behavior on longer tasks is waiting for third-party tests.

Speed will be a differentiator. It will not be the endgame.


Sources:

Source: https://www.inceptionlabs.ai/blog/introducing-mercury-2-5