Tokenizer has long been the unspoken rule of LLMs—everyone uses it, nobody publicly questions it. Then Nature published this last week:
Training large language models directly on bytes, without retraining
The goal is aggressive: take an existing token-level LLM and convert it to consume byte sequences without modifying the pretrained weights. Nobody has attempted this before.
What the paper actually does
The core method is retrofit—modifying an existing model rather than pretraining from scratch. That distinction matters.
Earlier byte-level efforts—ByT5, MegaByte, CANINE—all share the same problem: you have to redo pretraining. Training a GPT-4-class model costs upward of $100 million, and nobody wants to gamble that kind of money just to "try bytes."
Nature's approach: add a byte-level adapter on top of an existing model, freeze the original parameters, and train only the newly inserted modules. The core trade-off is straightforward:
- No retraining required, cutting retrofit cost by an order of magnitude
- Adapter parameters still need training, so small-scale fine-tuning is unavoidable
- Output quality is slightly lower than native token models, with a gap in the 1–2% range
The numbers: across multiple standard benchmarks, the retrofitted model hits roughly 98% of the original's performance while gaining all the benefits of byte-level modeling.
Why a Nature paper is tackling the tokenizer problem now
BPE and WordPiece subword schemes have been the standard for years, but the accumulated problems are significant:
OOV issues. BPE theoretically has no out-of-vocabulary tokens, but in practice, emoji, rare characters, and uncommon proper names often get split into single bytes or garbled fragments that the model cannot interpret.
Multilingual inequality. English averages about 4 characters per token, while Chinese sits around 1.5, and some languages fall below 1. Given equivalent corpus sizes, Chinese training is inherently penalized—the tokenizer works against it.
Special character chaos. @, #, URLs, and code snippets are treated as "junk" by tokenizers and frequently get chopped into meaningless fragments. Long-time GitHub Copilot users know this pain well: the model regularly breaks down on long URLs.
Engineering debt. Every model lab maintains its own special token list—custom delimiters, control symbols, and the like. Maintaining this is tedious, and one careless fine-tune can break compatibility.
A top-tier journal like Nature rarely publishes pure NLP engineering retrofits. The fact that this one made it signals something: byte-level modeling is moving from "fringe research" to "mainstream legitimacy."
Where the Hacker News discussion is stuck
The paper hit 23 points and 0 comments on Hacker News. The score suggests early readers endorse the quality; the silence means most people are still reading the paper rather than debating it.
But the technical sticking point is clear: sequence length explosion in byte-level inference.
Do the math:
- 4 English characters ≈ 1 token (typical BPE)
- Same 4 characters ≈ 12–16 bytes (UTF-8)
- Sequence length inflates 3–4×
- KV cache grows proportionally
- Memory scales linearly; attention is O(n²)
A 100K-token context window becomes a 300–400K-byte sequence. Inference cost on an H100 quadruples, and latency rises with it.
Byte-level models aren't production-ready for long contexts yet. The paper only reports short-text benchmarks; long-context behavior will need community reproduction.
Which of the top model labs is most likely to follow
A look at the three major players' positions:
Anthropic. Claude's tokenizer has always been relatively conservative, and the team has a history of BPE modifications (they've adjusted vocab size and special token design before). Their interest in byte-level should be highest, and their engineering culture can absorb the retrofit's tech debt. Most likely to follow within 6–12 months.
Google. The Gemini tokenizer team is among the strongest in the industry—they contributed SentencePiece. A byte-level adapter is "a new use of existing tools" for them. Follow probability: 60–70%.
OpenAI. GPT-5 architecture details are undisclosed. OpenAI is more likely to ship a stable version first and revisit byte-level for GPT-5.5 or GPT-6. Short-term follow probability is low.
Meta. Llama 4 is going the native multimodal route, pushing tokenizer concerns into unified multimodal representations. Meta is more likely to use plug-in patches (e.g., specialized preprocessing modules for URLs or code) than a full byte-level retrofit.
Contrarian take: byte-level isn't the endgame—tokenizers will return in a different form
My view diverges from the mainstream: byte-level is not the destination—it's a transitional solution.
The reasoning is simple: byte-level solves the "input side" problem (letting the model see raw data), but the "output side" remains a nightmare.
Byte-level generation means:
- Drastically lower bytes-per-second output, hurting user experience
- Constrained decoding becomes impractical for long-form generation
- Code generation can barely produce structured outputs (JSON, code blocks would need to be redesigned)
The output side still needs an aggregation unit—neither character nor byte, but some kind of "learned patch." My prediction for the next 1–2 years:
Learned byte-level patch: an intermediate form between byte and token, where the model learns to dynamically cluster adjacent bytes into meaningful patches.
This moves "segmentation" from preprocessing into the model itself, with dynamic decisions. One of NVIDIA, Anthropic, or xAI will likely publish the first work on this.
Editor's notes
Why does this matter? Not because "tokenizers are dying," but because a five-year-old architectural default is being formally challenged by a Nature-tier journal for the first time. It's a signal of architectural reflection within the LLM field.
Contrarian angle: the mainstream narrative says "byte-level is the future," but I believe byte-level will soon hit a hard wall on the output side. The real endgame is a model that learns to segment dynamically—not the static rules of BPE, nor a pure byte stream, but a dynamic learned patch. This won't mature until around 2027.
What makes this different from past attempts? ByT5 (2021), MegaByte (2023), and CANINE (2021) were all byte-level efforts that failed to dislodge BPE. The difference now: Nature's endorsement plus a retrofit approach that brings cost to an acceptable range. Byte-level is moving from "research demo" to "engineering option" for the first time. But the HN score of 23 with 0 comments also reminds us the engineering community is still digesting—this won't ignite overnight.
Open uncertainty: the paper provides no data on byte-level performance at 100K+ context lengths. If that breaks down, byte-level won't reach production anytime soon and the conversation will cool off.