Want to build a voice AI agent but keep getting stuck on the sequential latency of STT → LLM → TTS? The moment a user interrupts, your code falls over. Pipecat is the cleanest framework I've come across for stripping that complexity out of your business logic. With 14,555 stars on GitHub, maintained by Daily and the community, here's how it actually works.
What Problem Does It Solve
Real-time voice agents run into two common bottlenecks:
- Sequential waiting: STT finishes before LLM starts, and LLM waits for its first token before TTS begins. The three stages combined easily push latency past two seconds, and the user experience feels sluggish.
- Barge-in: When a user interrupts mid-response, you have to immediately stop TTS, flush the generated audio buffer, and feed the new question back into the LLM. Rolling your own solution means writing a pile of state machines.
Pipecat's answer is to wrap each step as a pluggable FrameProcessor and chain them into a Pipeline. The Transport layer (WebRTC, WebSocket, mini-program) only handles I/O. The Context aggregator only handles conversation history. Your business logic only handles "what do I want the model to say." Time-to-first-token lands in a range that feels natural.
How to Use It
Install:
pip install pipecat-ai[cli]
pipecat init quickstart
pipecat init scaffolds a runnable project and can hand off to Claude Code or Codex to generate code. The core of a 20-line voice bot looks like this:
# Minimal Pipecat example: interruptible voice bot
from pipecat.frames.frames import LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.cartesia.tts import CartesiaTTSService
from pipecat.transports.services.daily import DailyParams, DailyTransport
async def main():
transport = DailyTransport(
room_url="...", token="...",
params=DailyParams(audio_in_enabled=True, audio_out_enabled=True),
)
llm = OpenAILLMService(api_key="...", model="gpt-4o")
context = llm.create_context_aggregator({"messages": [{"role": "system", "content": "你是个助手"}]})
pipeline = Pipeline([
transport.input(), # WebRTC audio input
DeepgramSTTService(api_key="..."), # STT
context.user(), # User message aggregation
llm, # LLM
CartesiaTTSService(api_key="..."), # TTS
transport.output(), # WebRTC audio output
context.assistant(), # Assistant reply aggregation
])
task = PipelineTask(pipeline, params=PipelineParams(allow_interruptions=True))
await PipelineRunner().run(task)
Note allow_interruptions=True—that's the interrupt switch. context.user() and context.assistant() are two FrameProcessors that bundle STT text into LLM messages and feed LLM replies back into history.
How It Works (Architecture)
The core abstraction is a chain of FrameProcessors. A Frame is the data packet flowing between processors. The common ones:
StartFrame/EndFrame: Pipeline lifecycleTextFrame: text fragmentAudioRawFrame: raw audio frameLLMFullResponseStartFrame/LLMFullResponseEndFrame: LLM full-response boundaries
The pipeline scheduler pushes Frames to each processor in order. Whoever cares, handles it. A typical flow:
WebRTC Audio → STT → TextFrame → Context(user) → LLM → Token → TTS → AudioRawFrame → WebRTC Out
Transport only handles I/O (WebRTC in/out, mini-program bridging). Context aggregators only handle conversation history. LLM, STT, and TTS are all pluggable Services. The framework absorbs the business-logic complexity.
Multi-agent isn't a scheduler—it's several Pipeline composition patterns:
- handoff: One Pipeline completes, then passes its context to the next (e.g., a booking agent hands off to a weather-check agent).
- parallel fan-out: Run multiple Pipelines on the same input in parallel; pick the fastest or most appropriate reply.
- sidecar: Attach a Pipeline alongside the main one for monitoring, translation, or sentiment analysis without affecting the primary flow.
All three patterns support cross-process deployment, meaning you can run different agents in different containers.
Where It Shines / Where It Doesn't
The good:
- Comprehensive ecosystem: Client SDKs across JS, React, React Native, Swift, Kotlin, C++, and ESP32. Frontend devs get a Voice UI Kit with ready-made components.
- Built-in Pipecat Flows for structured conversation state management—form-style multi-turn dialogs without writing your own state machine.
- Transport layer supports Daily WebRTC, WebSocket, mini-programs, and more.
- CLI tool
pipecat initscaffolds in one command and integrates with Claude Code / Codex. - STT, LLM, TTS, and VAD are all swappable. Adapters exist for Deepgram, Cartesia, OpenAI, Anthropic, ElevenLabs.
- Commercially maintained by Daily, so it's not going anywhere.
The bad:
- Python-only on the server side. For client integration, you have to use the SDKs; Python doesn't run on mobile.
- No built-in defaults for STT, TTS, or LLM—you bring your own API keys.
- Docs are thorough but mostly in English. Main repo: pipecat-ai/pipecat.
- Steep learning curve if you've never done real-time audio. Just wrapping your head around the Frame lifecycle takes time.
I Tried It
Cloned the repo and ran examples/foundational/07-interruptible.py. Local startup takes about 8 seconds (mostly dependency install). After WebRTC connects, perceived conversation latency runs 600-800ms.
With allow_interruptions=True enabled, user speech stops TTS and triggers a fresh LLM inference within 200ms. PipelineTask cancellation and context cleanup both behave—no leaks.
The bottleneck is the LLM inference itself, not the framework. Translation: if it feels slow, optimize model choice and server-side TTFB before touching the Pipeline structure.
Who Should Use It / Who Shouldn't
Good fit:
- Teams building voice customer service, AI tutoring, or voice assistant products
- Complex dialog systems that need multi-agent handoff (e.g., triage before prescription)
- Anyone wanting a cross-platform (JS / iOS / Android / mini-program) client story
- Teams that want to control their STT/TTS stack and avoid vendor lock-in
Bad fit:
- Simple chatbots—overkill
- Pure-text scenarios—LiveKit Agents or a direct API call is lighter
- Tight budgets that can't absorb STT/TTS service costs (consider managed options like Vapi or Retell)
Editor's Take
What makes Pipecat worth watching isn't the 14.5k stars—it's that the team got the Pipeline abstraction right. They pulled the hard parts of voice agents (interruptions, streaming, context) out of business logic and turned them into framework primitives. LiveKit Agents and Agora each had their own approach. Pipecat is the first to standardize this layer in open source.
My read: in the second half of 2026, more vertical voice AI companies will build on Pipecat rather than rolling their own WebRTC. It levels the playing field on real-time audio engineering, shifting the moat to product and data.
What's still uncertain: support for end-to-end voice models (GPT-4o realtime, Gemini Live) is still catching up. Once those go mainstream, the three-stage STT/LLM/TTS pipeline abstraction itself might get bypassed—an end-to-end model doesn't need STT or TTS at all. Pipecat's next move is probably to generalize the Frame layer to adapt to any voice model, not keep optimizing three-stage scheduling.