Sander Dieleman posted a new piece on his personal blog last week titled Continuous Diffusion Language Models. The short version: today's LLMs all "guess one token at a time," and he wants to skip the token step entirely, running diffusion directly in continuous vector space.
The post lives here: sander.ai/2026/08/24/continuous-dlms.html. It's technical, but the core argument isn't complicated. Here's what stood out after reading it through.
The Fundamental Difference Between CDLM and Conventional LLMs
Every major LLM today — GPT, Claude, Llama, Qwen — runs on the same pipeline:
- Slice text into tokens (usually BPE or SentencePiece, vocab sizes from 30k to 250k)
- Map each token to a vector
- Predict a probability distribution over the next token autoregressively
- Sample a token, append it, repeat
This pipeline carries a few baked-in assumptions:
- Vocab size is a hyperparameter requiring tradeoffs
- Context length is bounded by attention's O(n²) complexity
- KV caching only works because of the "one token at a time" order
- Any parallel decoding attempt (speculative decoding, Medusa) is patching this paradigm
CDLM swaps out steps 1 and 4 together: no tokenization, generation happens via diffusion in a continuous vector space. Think Stable Diffusion for images — start from Gaussian noise, iteratively denoise — except the "noise" is a sequence of continuous vectors rather than image pixels.
Dieleman argues this isn't just "another flavor of diffusion" but a paradigm-level shift: tokenization is itself an inductive bias that assumes language is a discrete, countable sequence of symbols. Drop that assumption and the model learns different representations, with different degrees of generative freedom.
How much different? He doesn't give numbers — and as we'll see, that's a real problem.
Who Is Sander Dieleman?
Lots of people blog about ML. Dieleman isn't one of them. Quick background:
- Now: Research scientist at OpenAI (joined in 2024 from DeepMind)
- Previously: Led the audio team at DeepMind, drove multimodal embedding work for Gemini 1.5
- YouTube: 16k subscribers, deep learning tutorials with a strong reputation in the ML community
- Notable papers: YouTube-8M dataset, DeepMind's audio understanding series, Gemini multimodal work
He's credited on the Gemini 1.5 multimodal embedding paper for actual embedding work, not a courtesy authorship. That means firsthand engineering experience in "how to put different modalities into the same vector space."
CDLM isn't a hobby project. That matters for what's coming next.
Why This One Is Worth Watching
On its own, the post doesn't contain much that's strictly new — continuous diffusion has been kicked around for a year or two. But three signals together make it stronger:
- Diffusion has already proven out on images and video. Stable Diffusion 3, Flux, Sora — all continuous diffusion. The modality isn't the bottleneck; feasibility is established.
- Discrete diffusion LLMs already exist. LLaDA (Renmin University), JieyuZ2 and others have shown you can train diffusion in discrete token space — it just doesn't beat autoregression yet. That's the discrete exploration.
- Now jumping straight to "continuous." This skips past tokenization's fundamental limits. Moving from discrete to continuous diffusion means rewriting the loss function, the sampler, and the training pipeline. That's not a casual "let's try this" move.
Three signals stacked together reads more like a serious push in a direction than riding the diffusion hype wave.
Dieleman also engages with what generation looks like after training — preferences in the sampling process, degrees of freedom, fundamental contrasts with autoregression. This isn't the "diffusion is hot, let me publish something" flavor of paper.
The Practical Problem: Training Cost and Inference Speed
To be blunt, there are no numbers in this post.
No benchmarks, no loss curves, no comparison experiments. The HN discussion (item?id=49502611) currently sits at 11 points and zero comments — nobody has reproduced anything, and there's no public code.
That means two things:
- Short term, this won't ship. Continuous space means the loss function, sampler, and training infrastructure all need rewriting. Existing GPU kernels (TensorRT-LLM, vLLM, FasterTransformer) are optimized for discrete tokens plus autoregression. Switching paradigms means rebuilding from scratch.
- Long term, it could be disruptive. If training can be made cheaper than autoregression, the game changes. Today's work — optimizing KV cache, FlashAttention, speculative decoding — is all squeezing the current paradigm. The next paradigm shift could obsolete all of it.
Dieleman himself acknowledges this is early. He's arguing "why this path is worth walking," not "we've already walked it." That's responsible writing — but it also means anyone (including me) drawing strong conclusions is jumping the gun.
Editorial Take
The genuinely interesting move here isn't "diffusion for text." It's the claim that tokenization might not be necessary.
If continuous diffusion works, every optimization we have today — vocab size, context length, KV cache, FlashAttention, speculative decoding — becomes a local optimum. Every effort to improve transformers has been patching a paradigm that could have been bypassed.
My read: this won't threaten GPT or Claude in the short term. Three reasons:
- Infrastructure debt is massive. The entire inference stack needs rebuilding, and whoever moves first pays.
- No benchmark exists. There's no "MMLU for continuous diffusion LLMs" that the community agrees on.
- Ecosystem lock-in. Tons of downstream tooling (prompt engineering, function calling, agent frameworks) assumes a discrete token interface.
The contrarian angle: if this path does work, the cost of paradigm migration is bigger than people expect — not because the new model is stronger, but because the old ecosystem is expensive to move. What we treat as "infrastructure" today might just be "scaffolding from the autoregressive era." The next paradigm shift won't start from improving transformers. It'll start from going around them.
The uncertainty: I haven't seen any experimental data, so "can this even be trained" remains an open question. Dieleman's argument is mostly theoretical, and engineering implementation has plenty of places to go wrong. But the direction is worth watching — especially if OpenAI shows any related movement in the next six months, the signal gets loud.