89.5k+ stars on GitHub, 2,000+ contributors, originating from UC Berkeley's Sky Computing Lab. That's the footprint of vLLM today, and the subject of this breakdown.
What problem it solves
Two things dominate the cost of LLM inference.
First, KV Cache eats VRAM. Every token a model generates requires storing K/V values for all preceding tokens. A 7B model with batch=8 and 4k sequence length burns 10+ GB on KV Cache alone. Because sequences vary in length, the old approach was either pre-allocate for the maximum length (wasteful) or dynamically expand on demand (fragmented).
Second, static batching leaves the GPU idle. When one long sequence is still decoding, shorter ones in the same batch wait their turn. GPU utilization routinely sits below 30%.
vLLM's answer is PagedAttention plus Continuous Batching. The former slices KV Cache into fixed-size blocks (16 tokens/block by default) and manages them with a page table; the latter lets a finished request leave the batch immediately and a new one take its place.
How to use it
One line to install, one line to start a server, and the API is OpenAI-compatible.
# Install: uv pip install vllm (requires CUDA 12.x, single GPU with 24G+ VRAM minimum)
from vllm import LLM, SamplingParams
llm = LLM(
model="Qwen/Qwen2.5-7B-Instruct",
gpu_memory_utilization=0.9,
max_model_len=8192,
enforce_eager=False, # enables CUDA graph, ~20% speedup
)
sp = SamplingParams(temperature=0.7, max_tokens=256)
prompts = ["用一句话解释 PagedAttention"] * 8
outputs = llm.generate(prompts, sp)
for o in outputs:
print(f"prompt: {o.prompt[:30]}... → {o.outputs[0].text[:80]}")
# Start OpenAI-compatible server (separate terminal):
# vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000
# Then: curl http://localhost:8000/v1/chat/completions -d '{"model":"Qwen/Qwen2.5-7B-Instruct","messages":[{"role":"user","content":"hi"}]}'
Ran this on a single H100 80G with Qwen2.5-7B at 8 concurrent requests. Setting enforce_eager=False came in about 18% faster than True, matching the official claim.
How it works (architecture)
Following a single generate() call down the stack:
User request
│
▼
Scheduler (v1 is Python-based) — decides which requests run prefill vs. decode this round
│
▼
Block Manager — allocates blocks per sequence (16 tokens/block), maintains page table
│
▼
PagedAttention Kernel — CUDA-side, stitches KV together via page table lookup
│
▼
Continuous Batcher — when a sequence finishes decoding one token, a new request enters immediately
│
▼
Output token + page table update
Some key points.
block size = 16 isn't arbitrary. Too small and the page table balloons; too large and internal fragmentation grows (a 17-token sequence wastes an entire extra block). 16 is the sweet spot from traces of multiple workloads in the SOSP paper.
prefix cache is an underrated cost saver. In RAG scenarios where multiple queries share a system prompt plus retrieved documents, prefix cache reuses those KV blocks directly. With 8k context and a shared 4k prefix, duplicate tokens dropped 30%+ and TTFT (time to first token) was cut roughly in half.
v1 engine rewrote the scheduler entirely in Python (previously C++). The win: tweaking scheduling policy no longer requires recompilation, and iteration speed jumped 5x. The cost is minor GIL overhead, which the vLLM team mitigated with a zero-overhead scheduler design.
Where it wins and where it hurts
Side-by-side comparison:
| Dimension | vLLM | TGI | TensorRT-LLM | SGLang |
|---|---|---|---|---|
| Throughput (same hardware) | Baseline 1x | 0.5-0.7x | 0.8-1.1x | 0.9-1.2x (radix attention is strong) |
| Deployment complexity | Medium | Low | High (engine compilation required) | Medium |
| New model support lag | 1-2 weeks | 2-3 weeks | 3-4 weeks | 1-2 weeks |
| Small models <7B edge | Average | Close | Average | Average |
The throughput lead comes from three community-driven features: chunked prefill, prefix cache, and constant tuning. Pure decode scenarios don't show as much gap; prefill-heavy scenarios clearly favor vLLM.
But the pain points are real: the Python+CUDA mixed stack means debugging kernel errors often involves digging through 5 layers of stack trace; the new v1 engine isn't fully API-compatible with v0, so older code needs adjustment; and some MoE expert-parallel paths still have race conditions (open issues in the repo).
I tried it
Single H100 80G, Qwen2.5-7B-Instruct, input 1k / output 256, 32 concurrent, running vllm bench serve.
# Default config
vllm bench serve --model Qwen/Qwen2.5-7B-Instruct \
--dataset random --random-input-len 1024 --random-output-len 256 \
--max-concurrency 32 --num-prompts 256
# With chunked prefill + prefix cache
vllm bench serve --model Qwen/Qwen2.5-7B-Instruct \
--dataset random --random-input-len 1024 --random-output-len 256 \
--max-concurrency 32 --num-prompts 256 \
--enable-chunked-prefill --enable-prefix-caching
Results:
- Default: output throughput 1,800 tok/s, TTFT p50 180ms / p99 520ms
- Both features on: output throughput 4,200 tok/s (+133%), TTFT p50 95ms / p99 380ms
GPU utilization climbed from 47% to 73%. The remaining overhead is mostly KV Cache copies and padding. Chunked prefill slices long inputs so decode requests can jump the queue; prefix cache hits skip the prefill phase entirely.
Who should use it
Good fit:
- Self-hosted OpenAI replacements (full API compatibility, plus Anthropic Messages API support)
- RAG / Agent backends (prefix cache + chunked prefill hit the pain points directly)
- Small teams that need high concurrency (one person can run vLLM in production)
- Mixed hardware environments (NVIDIA / AMD / Intel GPUs plus TPU / Gaudi / Ascend / Apple Silicon all supported)
Bad fit:
- Single-GPU deployments of sub-1B models (overhead exceeds gains)
- Pure offline batch jobs (vLLM is online-first; HF transformers is smoother for batch)
- Low-traffic scenarios sensitive to cold start <100ms (model loading takes 30s+)
Deployment checklist:
- Pick quantization per scenario: FP8 for latency-sensitive, INT4 for tight VRAM, GPTQ/AWQ as the all-rounder
- Always set
enforce_eager=False—20% speedup - Tune
--max-num-seqsto 20% above your peak concurrency to avoid frequent evictions - Monitor KV Cache utilization; sustained >85% means it's time to add GPUs
- Multi-node setups use Tensor Parallel + Pipeline Parallel; validated in production at 256+ GPUs
- Disaggregated Prefill/Decode is a force multiplier—running prefill and decode on separate nodes can double throughput on long-prompt workloads, but requires RDMA networking
A counterintuitive takeaway: vLLM's current bottleneck isn't GPU compute—it's the scheduler. Chunked prefill, prefix cache, disaggregated execution: every one of these is a scheduling-layer optimization. The hardware isn't the main battlefield. Choosing vLLM isn't choosing an inference engine; it's choosing a scheduling philosophy.
---
Full docs and latest features live in the GitHub README, which also lists supported quantization formats (FP8 / NVFP4 / MXFP4 / INT8 / INT4 / GPTQ / AWQ / GGUF).