Frameworks

Building a Low-Latency Voice Agent with TEN Framework

TEN Framework is an open-source real-time multimodal conversation stack with 11k+ GitHub stars, bundling VAD, turn detection, and Docker self-hosting. It sits between LiveKit's RTC focus and Picovoice's component-level approach.

Font size

TEN Framework in Practice: Building a Low-Latency Voice Agent from Scratch

ten-framework on GitHub has crossed 11,120 stars. It's an open-source framework for real-time multimodal conversational AI, with VAD, turn detection, and memory modules built in, plus Docker-friendly self-hosting. I ran it end-to-end: the voice_assistant example was up in five minutes, with perceived latency around 600-800ms. Here's my assessment and benchmark notes.

What Problem It Solves

Real-time multimodal conversational AI runs into four recurring pain points:

  • Latency: end-to-end needs to stay under 500ms to feel human.
  • Streaming: audio is continuous, so waiting for a full utterance before processing kills responsiveness.
  • Interruption and silence: the AI has to know when to talk and when to hold back.
  • Multimodal chaining: audio in, text processing, audio out — the data flow has to stay connected.

A typical LLM wrapper turned into a voice agent will trip on the first three. VAD thresholds are misconfigured and the bot either cuts in or leaves dead air. Streaming is done wrong and latency climbs past a second. Turn-taking relies on a crude silence timeout.

TEN folds VAD, turn detection, and multimodal pipeline into a single framework — not another agent SDK. It positions itself as a "complete toolchain for real-time multimodal conversational agents," rather than a general-purpose orchestrator like LangChain.

How to Use It

Clone, install, run the example. Three steps:

git clone https://github.com/TEN-framework/ten-framework.git
cd ten-framework/agents/examples/voice_assistant
pip install -r requirements.txt

Configure API keys for ASR / LLM / TTS (OpenAI, ElevenLabs, etc.), then:

python app.py

Once the service is running, open localhost:8080 in a browser, connect over WebSocket, and start talking. I had it running in five minutes without hitting any snags.

The core code is short:

from ten import Agent

# Initialize a voice agent: ASR → LLM → TTS pipeline
agent = Agent(
    name="voice_assistant",
    vad="ten_vad",              # Voice activity detection
    turn_detection="ten_td",    # Turn detection, prevents interruptions
    asr="whisper",              # Speech-to-text
    llm="gpt-4",                # Response generation
    tts="elevenlabs",           # Text-to-speech
    memory="buffer:10"          # Keep last 10 turns of context
)

# Start WebSocket server; browser mic streams in directly
agent.serve(transport="websocket", port=8080)

Graph-based orchestration: nodes are modules, edges are data flows, declarative configuration, no custom scheduler required.

How It Works

The architecture is graph-based agent orchestration:

  • Nodes: ASR, VAD, turn detection, LLM, TTS, memory.
  • Edges: data flows — audio frames and text tokens move between nodes.

The data flow looks like this:

  1. Audio arrives → VAD detects speech.
  2. Turn detection decides whether to respond.
  3. ASR converts speech to text.
  4. LLM generates a reply.
  5. TTS synthesizes audio.
  6. Audio streams back to the client.

The key design choice is turn detection. The module is called ten-turn-detection, and it goes well beyond a VAD threshold that measures "how much silence equals a finished sentence." It weighs semantic pauses, prosody, and context to decide when the AI should speak. I held a deliberate two-second pause; the AI didn't jump in. I spoke quickly back-to-back; the AI didn't drop anything.

Two transport modes are supported: RTC (WebRTC) and WebSocket. WebSocket suits local debugging and simple scenarios; RTC fits production environments that need end-to-end low-latency transport.

Strengths and Weaknesses

Strengths:

  • 11k+ stars, active community.
  • README available in seven languages: English, Simplified Chinese, Japanese, Korean, Spanish, French, Italian.
  • Docker self-hosting is straightforward — docker compose up brings the stack online (first image pull takes about 10 minutes).
  • Coherent ecosystem: five repos — TEN Framework, Agent Examples, VAD, Turn Detection, and Portal — all maintained in-house.
  • Rich examples: voice_assistant, doodler, speaker-diarization, covering voice assistants, drawing assistants, and multi-speaker separation.

Weaknesses:

  • Python-only language binding. Backend teams on Go or Rust have to wrap it themselves.
  • Smaller ecosystem than LiveKit, which has years of WebRTC depth.
  • Documentation is English-first. Simplified Chinese exists, but deep technical docs stay in English.
  • Chinese ASR/TTS vendors (iFlytek, Alibaba, Volcengine) require custom adapters.

Compared to LiveKit Agents:

  • LiveKit Agents leans toward real-time communication infrastructure — WebRTC is its strong suit.
  • TEN focuses on the full "multimodal conversational agent" scenario out of the box.
  • If LiveKit is already in your stack, migration cost is high. If you're starting from zero, TEN is the simpler path.

Compared to Picovoice:

  • Picovoice is a voice components vendor — VAD, wake words, ASR.
  • TEN adds a complete pipeline on top, ready to run.

Hands-On Test

Ran the voice_assistant example: local service plus browser mic.

Latency feel: End of speech to AI reply lands around 600-800ms. Faster than a pure cloud API stack (1s+), slower than end-to-end voice models like the Realtime API (200-300ms). Good enough, but not the fastest option.

Turn detection: Held a two-second pause; the AI didn't jump in. Spoke back-to-back quickly; nothing was missed. A simple VAD threshold can't do this — the turn detection module is doing real work.

Docker deployment: docker compose up brings the stack up cleanly, but the first image pull takes about ten minutes, including model weights. Production deployments should pre-pull images.

Code size: The example directory is around 200 lines, with about 50 lines of core logic. A new developer can read and run it without consulting docs.

Who Should Use It (and Who Shouldn't)

Use it if:

  • Your team wants to self-host a real-time voice agent.
  • Latency under one second matters.
  • You don't want to build VAD and turn detection from scratch.

Skip it if:

  • You only need speech-to-text (Picovoice is lighter).
  • You're already on LiveKit Agents and don't want to migrate.
  • You depend on Chinese ASR providers but don't want to write adapters.

Good fit for:

  • Customer support automation.
  • Voice assistants.
  • Education tutoring.
  • Real-time translation.

Editorial Take

TEN positions itself as a "complete toolchain for real-time multimodal conversational agents," not another LLM orchestration framework. It tackles the voice-specific problems of latency, streaming, and turn-taking — areas where LangChain and LlamaIndex don't help, because their design assumption is "text in, text out," with no awareness of audio streams or time sensitivity.

The split with LiveKit comes down to "agent-first" vs "RTC-first". TEN puts conversational logic front and center; LiveKit puts audio/video transport front and center. If the core requirement is "let the AI hold a conversation like a human," TEN targets the right problem. If the core requirement is "AI joining a multi-party video call," LiveKit is the lower-level foundation.

What's different about TEN compared to past open-source voice frameworks: it packages the "dirty work specific to voice agents" — VAD and turn detection — into ready-to-use modules, instead of pushing threshold tuning onto the user. Older voice-agent stacks required reading Silero VAD docs, tuning silence timeouts, and handling partial ASR results manually. TEN bundles all of that.

Prediction: 2026 will see more "vertical-scenario agent frameworks" — for customer service, education, medical intake, and similar domains. TEN is the early signal in that race: a proof point that "a complete pipeline for a vertical scenario" beats "a general-purpose orchestration framework" on user value.

Open questions: TEN's Chinese-language community is still early, and Chinese ASR/TTS adapters have to be hand-rolled — that's both a barrier and an opening. An official set of vendor adapters would accelerate adoption in the Chinese market significantly. The Python-only binding is another uncertainty; once Rust or Go support lands, production-readiness takes a meaningful step up.

Reference: ten-framework on GitHub

Source: https://github.com/TEN-framework/ten-framework