Opinion

LLM Voice: The Hard Parts of Real-Time Voice Agents

Martin Fowler examines how to build LLM-powered, real-time voice applications. The core challenge is not combining ASR, LLM, and TTS, but controlling end-to-end latency, interruptions, and system state.

Font size

Martin Fowler's article LLM Voice is not the announcement of a new model. It addresses a more specific and difficult question: how should an LLM be incorporated into a real-time voice application?

The article reached the front page of Hacker News, where it earned 210 points and 123 comments. Those numbers show that developers are paying attention, but discussion activity does not mean the solution is mature. If anything, the implementation costs and architectural disagreements in the comments reveal how far voice Agents still are from becoming dependable products.

Original article: https://martinfowler.com/bliki/LLM-Voice.html
Hacker News discussion: https://news.ycombinator.com/item?id=50290177

Not a New Model Release

Fowler did not release a new speech model or introduce a new benchmark. His article is an exploration of architecture and engineering practices, with a focus on streaming, latency, interruption handling, and state management.

This distinction matters.

Voice capabilities can come from a combination of ASR, LLM, and TTS, or from an end-to-end speech model. But a product's usability does not hinge on whether its final response is correct. If a user speaks and then waits two seconds for an Agent to begin responding, or says “stop” while the assistant continues reading the entire response, the experience is more damaging than the occasional inaccurate word.

LLM Voice therefore cannot be evaluated solely through model parameters or text question-answering scores. Real-time voice involves an entire system pipeline, and every stage can introduce additional delay, dropped words, or state errors.

Conversation Ability Is Not the Same as Usability

A typical prototype follows this sequence:

  1. Record audio.
  2. Send it to ASR to produce text.
  3. Pass the text to an LLM.
  4. Send the response to TTS.
  5. Play the generated audio.

The code works and the demo is straightforward, but it resembles a recorded-message pipeline rather than a real-time voice Agent.

There are three fundamental challenges:

  • End-to-end latency. After the user finishes speaking, the system needs to acknowledge receipt quickly and begin generating playable speech.
  • Streaming output. Waiting for a complete response before synthesizing it simplifies the architecture, but it weakens the sense of a live conversation.
  • Unexpected interruptions. Users may change the question, correct the assistant, ask it to pause, or simply repeat themselves. The Agent must recognize these actions rather than continue executing the previous task.

Connecting ASR, LLM, and TTS is only the plumbing. The experience depends on state control after those components are connected.

Architecture: Inside a Real-Time Voice Pipeline

A real-time voice pipeline can be divided into five layers.

1. Audio Capture

This layer must capture audio continuously instead of waiting for the user to finish before submitting it. A microphone usually produces frames of a fixed duration, such as 20 or 30 milliseconds each.

Capture does not mean sending every frame for recognition. Voice activity detection (VAD) can help determine when the user starts and stops speaking. Fixed thresholds, however, are prone to errors in noisy environments, during pauses, and at natural speaking speeds. A more reliable approach combines VAD, intermediate ASR results, and a server-side state machine.

2. Speech Recognition

ASR should ideally operate in streaming mode. The system needs to continuously emit partial transcripts so the application knows what the user has said and can detect interruptions as early as possible.

Final transcripts are then used to derive the user's complete intent. This creates a familiar trade-off: streaming recognition is faster but may revise earlier text as context changes, while segmented recognition can be more accurate but delays requests sent to the LLM.

Production systems often need both. Partial transcripts support real-time state, while final transcripts support reliable execution.

3. LLM Inference

The LLM can also stream tokens. The important objective is not merely to generate the first token quickly, but to produce the first playable speech segment quickly.

This requires a pipeline across the LLM, text normalization, TTS, and playback rather than serial waits. Once the first sentence is generated, it can be sent to TTS. As soon as TTS produces its first audio segment, playback can begin while subsequent sentences continue to be generated and synthesized.

This works well for short answers. If a response requires weather data, an order lookup, or a database query, the latency profile changes. The product must then choose between acknowledging the request first, such as “I'm checking that,” and calling the tool directly.

4. Speech Synthesis

TTS should support both streaming input and output. Synthesizing a complete text passage in one pass can produce stable audio quality, but the user must wait for the entire answer. Splitting the response by sentence enables earlier playback, although it can divide speech at awkward semantic boundaries.

A practical compromise is to segment by complete sentence. Saving a few dozen milliseconds is rarely worth producing fragmented-sounding speech.

5. Playback

The playback layer must support cancellation, flushing, and replacement. Without those capabilities, an old audio queue may continue playing even after the user has interrupted the Agent.

Echo and self-interruption create further complications. The assistant's voice from the speakers may be captured by the microphone and incorrectly detected as the user starting to speak. The system needs playback state, AEC, or another echo-suppression mechanism to prevent the Agent from interrupting itself.

The full pipeline can be summarized as follows:

microphone frames
  -> streaming ASR + VAD
  -> streaming LLM
  -> sentence chunks
  -> streaming TTS
  -> cancellable audio playback

This is not the only possible implementation, but it is considerably closer to a real-time voice product than recording once and producing one answer.

Engineering Challenges: Interruptions and State Management

Interruption handling is a hard threshold for voice Agents.

When the user starts speaking, the system may need to interpret at least one of three intentions:

  • The user is adding information.
  • The user is modifying or canceling the original request.
  • The user is starting an entirely new conversation.

These cannot all be treated as “stop playback.” The system needs to know the Agent's current state: listening, thinking, calling a tool, speaking, or awaiting confirmation.

A practical state machine might look like this:

IDLE -> LISTENING -> THINKING -> SPEAKING
  ^         |           |          |
  |         +---- USER_INTERRUPT ----+
  +-----------------------------------+

A real implementation, however, cannot rely on state names alone. Every interruption must also handle request IDs, playback generation IDs, and the lifecycle of tool calls.

Canceling Playback Does Not Mean Canceling the Request

When a user says “stop,” they may only want to end the audio output while the background query continues, or they may want to cancel the entire task. These cases require separate handling.

A safer approach is layered cancellation:

  • Cancel the current audio playback immediately.
  • Invalidate the current conversation turn.
  • Discard unconsumed LLM and TTS output associated with the old turn.
  • Decide independently whether external tool calls should be canceled.

Context Replay

Streaming ASR continually revises recognition results. By the time a new transcript arrives, the LLM may already have issued a request based on an older version or even begun calling a tool. Simply appending the new text to the conversation history would leave both the old and new questions in context.

A more reliable approach assigns an ID to each conversation turn and records the transcript version used for that turn. Once a revised recognition result replaces the previous version, only requests that remain valid are allowed to proceed.

For tool calls already in flight, the system must distinguish between “cancel the HTTP request” and “ignore the late result.” The latter is often overlooked: even after a tool finishes, the Agent may still insert its stale result into the new conversation.

Agent Tool Calls

A voice Agent introduces an additional layer of uncertainty compared with a text-only Agent. Users cannot easily see a long block of text or confirm which operation is being performed.

For actions involving payments, deletions, or sending messages, the system should provide a brief confirmation that the user can interrupt. If the user changes their decision after hearing it, the old confirmation state must be revoked rather than executed.

Tool calls do not always require complete silence. A brief statement such as “I'm checking your order” can reduce the anxiety caused by waiting, but that statement also has latency and may itself be interrupted. Whether to speak should depend on the tool's expected duration: operations that return within tens of milliseconds can run silently, while slower operations benefit from status feedback.

Evaluation: Look Beyond Response Accuracy

Fowler's article does not provide a definitive benchmark or quantitative performance data. Teams should therefore avoid assuming that a ready-made, universal evaluation standard already exists.

They can, however, begin by defining their own system-level metrics.

Time to First Audio

This is the time from the end of the user's speech to the start of playback of the first valid audio segment. It reflects the user's experience more closely than time to the first token.

Measurement requires explicit boundaries. Does the clock start when the final audio frame reaches the server or when the server detects that the user has finished speaking? Does it stop when the first TTS segment is written or when it actually plays through the speakers? Different definitions can produce dramatically different results.

Total Response Time

This measures the time from the end of the user's speech to completion of the spoken response. It is useful for evaluating long answers and tool calls, but it does not replace time to first audio.

Interruption Recovery Time

This is the time between the user's interruption and the point when old audio stops and the system enters the correct state. If the assistant continues speaking for three seconds after the user says “stop,” the metric has little value.

Tool Call Success Rate

At minimum, evaluation should distinguish among the following:

  • Whether the correct tool was selected.
  • Whether its parameters were correct.
  • Whether an interrupted action was incorrectly executed.
  • Whether a late result contaminated the current conversation.
  • Whether actions requiring confirmation were properly confirmed or canceled.

Speech and State Consistency

A correct final transcript does not guarantee a correct real-time interaction. Tests should also cover noise, repetition, competing speech, long pauses, network instability, and ASR transcript corrections.

A simple script can record key timestamps, while manual annotation can assess whether users remain willing to continue the conversation:

from time import perf_counter

start = perf_counter()
first_audio_at = None

# 在实际实现中替换为流式 TTS 的首个音频回调
for chunk in audio_chunks():
    if first_audio_at is None:
        first_audio_at = perf_counter()
        print(f"first_packet_ms={(first_audio_at - start) * 1000:.1f}")
    play(chunk)

This is not a complete evaluation framework, only a starting point for timing. A production system should also record request IDs, transcript versions, interruption events, and tool-call states; without those fields, it is difficult to determine which layer introduced the delay.

Product Boundaries for Voice Agents

Voice will not simply replace existing interfaces. Keyboards, buttons, and web pages remain useful because they support search, editing, scanning, and complex input. Voice is more likely to become an additional entry point for existing products than their sole interface.

What distinguishes this moment from earlier ASR assistants is that LLMs now participate directly in the real-time interaction loop. Older systems typically followed a “transcribe, query, announce” sequence. An LLM Voice system may instead understand, reason, call tools, and handle a user's changing mind simultaneously. System state evolves from a request chain into a continuously changing state machine.

The competitive advantage of voice Agents will not come only from natural-sounding voices. Time to first audio, interruption recovery, and isolation of late results are more likely to form durable product moats. Models can be replaced, but architecture and evaluation cannot easily be added at the last minute.

It remains uncertain whether end-to-end speech models will soon eliminate most of the problems inherent in ASR–LLM–TTS architectures. Even if they do, tool execution, permission control, confirmation mechanisms, and state management will remain in the application layer. Faster models will not automatically make request cancellation reliable.

Source: https://martinfowler.com/bliki/LLM-Voice.html