ByteDance Seed and Tsinghua AIR have jointly released DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization), an open-source system targeting large-model RL training in long Chain-of-Thought (CoT) scenarios. It landed on the HN front page with 10 points and is still gaining traction.
The repo at BytedTsinghua-SIA/DAPO ships more than just the algorithm — system, data, and configs are all laid out. Here's what I found digging through the code.
The Problem DAPO Solves
Anyone running LLM RLHF has hit these two walls:
- Long-horizon rollouts burn thousands of tokens per sample, so GPU time gets eaten alive by sampling.
- In math reasoning, hundreds of reasoning steps collapse into a single 0/1 reward, making PPO's advantage estimates extremely noisy.
The paper title spells it out: A Large-Scale Reinforcement Learning System for Long Chain-of-Thought. The main battleground is math reasoning benchmarks like AIME and AMC.
What they open-sourced is a full stack — system, algorithm, and data — not a standalone algorithmic tweak.
Two Key Differences from Traditional RLHF
Decoupled Clip
Standard PPO uses a single clip ratio to constrain policy updates. DAPO decouples the clip range into upper and lower bounds and treats them independently. In reward-sparse long-horizon tasks, the asymmetry in gradient clipping gets amplified, so splitting the bounds yields more stable training.
Dynamic Sampling Policy Optimization
Long-CoT rollouts are brutally expensive. If 80% of a batch is answered correctly (all rewards = 1) and only the remaining 20% carry gradient signal, most of the compute is wasted. Dynamic Sampling pre-rollouts every prompt in a batch, filters out samples that are all-correct or all-wrong, and keeps only those with meaningful gradient.
Intuitively, it's like pushing importance sampling's "important" step upstream into data selection.
Comparison
| Dimension | OpenRLHF | verl | DAPO |
|---|---|---|---|
| Focus | General RLHF | General RL + Agent | Long-CoT reasoning |
| Core algorithm | PPO / REINFORCE | PPO + variants | Decoupled Clip + Dynamic Sampling |
| Framework | DeepSpeed | Ray + vLLM | Modified verl |
| Open-source coverage | Fairly complete | Fairly complete | Full system stack |
DAPO isn't trying to be a general-purpose RLHF framework. It does one thing: make long-CoT RL training engineerable.
The ByteDance + Tsinghua Collaboration Model
The repo name BytedTsinghua-SIA says it all: ByteDance provides compute and engineering teams; Tsinghua AIR contributes the algorithmic ideas.
The joint-lab pattern between Chinese tech giants and top universities has been running since USTC-Microsoft and THU-IBM days. What's distinctive about DAPO is that ByteDance is open-sourcing industrial-grade RL training infrastructure directly — data pipelines, distributed rollout, reward services.
Behind this is a strategic bet. In the RL infra space, the ByteDance ecosystem is playing a "public utility" role akin to a Linux Foundation: get everyone building on your substrate, and the moat becomes data and application scenarios.
What the Code Tells Us
After cloning and poking around:
DAPO/
├── README.md
├── verl/ # Training framework, modified from ByteDance's verl
├── examples/ # Config examples
│ └── aime_mix_train.yaml
├── scripts/ # Launch scripts
├── reward_server/ # Reward service (deployed separately)
└── docs/ # Algorithm notes
A few observations:
1. The training framework is verl-based
The verl/ directory confirms DAPO is built on ByteDance's own verl (Volcano Engine Reinforcement Learning), which sits on Ray + vLLM and supports PPO, GRPO, and other algorithms.
DAPO adds two layers of customization on top:
- Algorithm layer: Decoupled Clip and Dynamic Sampling.
- System layer: decoupled reward service and rollout optimizations.
2. The data format is production-grade
Schema from examples/aime_mix_train.yaml:
data:
train_files: ~/data/rl/aime_mix/train.parquet
val_files: ~/data/rl/aime_mix/test.parquet
prompt_key: prompt
max_prompt_length: 2048
max_response_length: 8192
train_batch_size: 512
val_batch_size: 256
Notice max_response_length: 8192 — that's a textbook long-CoT setting. Standard RLHF tasks cap out around 2048.
3. Can a regular researcher actually run this?
Bar is high:
- At minimum 4-8 H100/A100 GPUs (the README suggests 16).
- Tens of GB of math problem data (AIME, MATH, NuminaMath).
- Ray cluster deployment know-how.
But the repo ships complete scripts/, and if you have the hardware, bash scripts/run_dapo.sh should theoretically get you off the ground. Compared to "open-sourced but unreproducible" releases, DAPO is on the generous side.
Who Should Care
Prioritized:
- Small reasoning-model teams: Qwen, DeepSeek, and Yi are already running long-CoT RL, but smaller teams lack the engineering muscle. DAPO gives them a relatively complete starting point.
- Anyone burned by GRPO/PPO: DAPO's dynamic sampling and decoupled clip are directly portable engineering lessons.
- RL infra investors and watchers: The ByteDance ecosystem (Volcano Engine + Seed) keeps doubling down on RL infrastructure — a signal for whether RL infra is forming into a standalone category.
- Academic researchers: With Tsinghua AIR backing, full code, and the paper, this is a textbook industrial-grade baseline.
Editorial Notes
ByteDance's open-source cadence around RL infra follows a clear logic: free infrastructure; the real moat is data and scenarios.
This differs from Meta's Llama playbook. Meta open-sourced model weights to counter OpenAI's API lock-in — the business case being "more people use my models → more people buy my GPUs/ads." ByteDance's DAPO, verl, and UI-TARS releases are closer to: make RL training accessible to everyone → RL becomes a utility → future revenue comes from compute and APIs.
What's specifically new this time? Three details:
- Engineering long-CoT RL used to be a closed-shop problem only OpenAI and Anthropic could solve. DAPO turns it into an open problem.
- Decoupled Clip + Dynamic Sampling isn't an algorithmic innovation — it's an engineering one. That tells you RL training's bottleneck is no longer algorithms but data sampling efficiency and GPU utilization.
- The repo naming
BytedTsinghua-SIAreflects a new industry-academia collaboration model — the giant provides compute infrastructure, the university supplies algorithmic ideas, and both sign off but own their respective strengths.
Open questions: I haven't reproduced DAPO's training cost or actual convergence speed yet. The README numbers might be optimistic. If you've run it, drop your real-world experience in the comments.
References: