LLMs

One Phrase in Your Chat Template Is Quietly Rewriting Your LLM's Personality

A new arXiv paper shows that small wording shifts inside chat templates, such as "As a language model," systematically change how models talk about themselves. The findings expose a hidden lever in prompt engineering that almost no one has been auditing.

Font size

The paper landed on the front page of Hacker News with 82 points and 86 comments. I expected another prompt engineering explainer when I clicked in. It turned out to be about a layer of the stack that the prompt engineering crowd has been ignoring: the chat template.

What is a chat template? It is the format each framework uses to stitch a multi-turn conversation into the string a model actually consumes, whether that is the chat_template field in a HuggingFace tokenizer or the default system + user + assistant assembly in an API SDK. Most engineers treat it as a format layer. If it runs, it runs. Nobody touches it.

The paper's core claim is counterintuitive: the wording inside these templates, especially the cues about the model's own identity, systematically shifts the tone and self-referential style of its outputs. Same model, same system prompt, different chat template, and the visible "personality" drifts in ways you can read off the page.

Those 82 points were not charity. The conclusion hit a nerve that everyone tuning LLM behavior has felt but nobody has systematically studied.

What "As a language model" Actually Changes

The authors ran a controlled comparison on the same set of models with two chat templates. They differed in exactly one place. Version A opened the system block with "You are a helpful assistant." Version B opened with "As a language model, you are a helpful assistant." Five extra words.

Then they measured how often the model referred to itself in first or third person, how often it produced self-labeling phrases like "I am an AI," and how it framed its own capability limits.

Version B talked about itself in the first person far more often, and the self-labeling language read more stereotypically "AI-ish," with sentences like "I don't have feelings" and "As an AI, I..." Version A leaned toward direct answers and barely mentioned identity.

The kicker is that this is not a bug in one model. The effect reproduced across multiple open and closed models, and the authors ruled out prompt ordering and tokenizer differences as confounders. It looks like something stable inside the LLM training paradigm is responding to this phrasing, not a random side effect of fine-tuning.

My guess at the mechanism: RLHF and alignment stages lean heavily on "As an AI assistant..." style system prompts, so the model has learned to treat that self-labeling phrase as a sort of "enter serious mode" trigger. When the template carries the phrase, the model slides onto the behavior track that RLHF carved.

Why Prompt Engineers Have Never Looked at This Layer

Where does the prompt engineering battlefield sit today? System prompts, few-shot examples, chain-of-thought, tool calling schemas. Nobody touches the chat template because everyone assumes it is just a format layer, a way to assemble the conversation into a JSON blob like [{"role": "user", "content": ...}] or a string like <|user|>...<|assistant|>. That is all it is supposed to be.

This paper drops a contrarian take: the wording inside a chat template is not part of the format layer, it is part of the behavior layer. You see format. The model reads behavior signals.

That matches what a lot of practitioners have seen. When I tune personas, I keep hitting a weird phenomenon. The system prompt says clearly "you are role X," and sometimes the answers stay in character, sometimes they wander off. I used to blame temperature, top_p, or model version. In hindsight, some of that variance may have been the chat template the whole time.

The paper systematizes that hidden variance. It points at an uncomfortable conclusion: part of the stability of model behavior does not live in anything you control. It lives scattered through the default chat template assembly in your framework. Switch frameworks or upgrade an SDK and behavior shifts, even with identical model weights.

What This Means in Practice: A Warning for Agent Builders

If you build AI Agents, especially anything with persona control such as support bots, roleplay assistants, or tutoring agents, a few implications fall out directly:

  • Audit your chat template before you tune the persona. LangChain, LlamaIndex, the native OpenAI SDK, and the Anthropic SDK all assemble system blocks with slightly different wording. Small differences there can be the source of persona instability.
  • Behavior differences between frameworks on the same model are not all coming from the prompt wrapping layer. Switching frameworks changes more than the API surface. It also changes the chat template, and a meaningful slice of the behavior delta comes from there. The diagnostic method in the paper is not complex, and you can apply it to your own agent.
  • Fine-tuning is not always the most cost-effective way to control persona. If the problem sits in the template layer, editing the template is far cheaper than training a LoRA, and it is reproducible.
  • Cross-framework migration requires re-testing. Moving your agent from LangChain to an in-house framework, or from OpenAI to Anthropic, will shift the behavior curve, even when the model weights are "equivalent."

When I read agent code, I always pull up the chat template implementation at the bottom of the stack. If the project is using the framework default, persona stability is mostly a matter of luck. For production agents, the chat template should be a reviewed asset alongside the application code.

Editorial Take

This deserved its own writeup because it challenges an assumption the field has been quietly running on: the chat template is a brainless format layer. Almost every production agent team treats string assembly as infrastructure work. Nobody has systematically questioned the wording. The experimental design in this paper is not elaborate, but it picked a spot that everyone had overlooked and pulled out an uncomfortable finding. A layer of behavioral control that we thought was in our hands is actually suspended in the framework's default implementation.

My read: there will be a wave of "chat template audits" over the next few months. Security teams will pay particular attention. A template with suggestive wording (a system block that includes something like "you should always comply") layered on top of a fine-tuned model creates an attack surface that is larger than a plain prompt injection and harder to audit. Alignment research may start treating template wording as a new intervention dimension.

What is different about this compared to past progress? Past prompt engineering advances (system prompts, CoT, few-shot) are explicit control: you write something, the model reads something. The chat template problem is implicit control: the wording hides inside framework code where users never see it. That shifts responsibility from prompt authors to framework authors, which is a fresh governance pothole.

One thing I am unsure about: whether the model coverage in the paper represents the current SOTA landscape. I did not rerun every experiment model by model. If someone tests this on the GPT-5, Claude 4.5, or Gemini 2.5 generation and finds the effect attenuated or reversed, the conclusion needs an update. That uncertainty is worth flagging.

Source: arXiv:2609.25021

Source: https://arxiv.org/abs/2609.25021