Frameworks

DAPO: How ByteDance and Tsinghua Open-Sourced Long-CoT RL Training

ByteDance Seed and Tsinghua AIR have open-sourced DAPO, a full-stack RL training system for long chain-of-thought reasoning built on top of verl. It tackles the engineering bottlenecks of long-context RL with decoupled clipping and dynamic sampling.

Font size

ByteDance Seed and Tsinghua AIR have jointly released DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization), an open-source system targeting large-model RL training in long Chain-of-Thought (CoT) scenarios. It landed on the HN front page with 10 points and is still gaining traction.

The repo at BytedTsinghua-SIA/DAPO ships more than just the algorithm — system, data, and configs are all laid out. Here's what I found digging through the code.

The Problem DAPO Solves

Anyone running LLM RLHF has hit these two walls:

  • Long-horizon rollouts burn thousands of tokens per sample, so GPU time gets eaten alive by sampling.
  • In math reasoning, hundreds of reasoning steps collapse into a single 0/1 reward, making PPO's advantage estimates extremely noisy.

The paper title spells it out: A Large-Scale Reinforcement Learning System for Long Chain-of-Thought. The main battleground is math reasoning benchmarks like AIME and AMC.

What they open-sourced is a full stack — system, algorithm, and data — not a standalone algorithmic tweak.

Two Key Differences from Traditional RLHF

Decoupled Clip

Standard PPO uses a single clip ratio to constrain policy updates. DAPO decouples the clip range into upper and lower bounds and treats them independently. In reward-sparse long-horizon tasks, the asymmetry in gradient clipping gets amplified, so splitting the bounds yields more stable training.

Dynamic Sampling Policy Optimization

Long-CoT rollouts are brutally expensive. If 80% of a batch is answered correctly (all rewards = 1) and only the remaining 20% carry gradient signal, most of the compute is wasted. Dynamic Sampling pre-rollouts every prompt in a batch, filters out samples that are all-correct or all-wrong, and keeps only those with meaningful gradient.

Intuitively, it's like pushing importance sampling's "important" step upstream into data selection.

Comparison

DimensionOpenRLHFverlDAPO
FocusGeneral RLHFGeneral RL + AgentLong-CoT reasoning
Core algorithmPPO / REINFORCEPPO + variantsDecoupled Clip + Dynamic Sampling
FrameworkDeepSpeedRay + vLLMModified verl
Open-source coverageFairly completeFairly completeFull system stack

DAPO isn't trying to be a general-purpose RLHF framework. It does one thing: make long-CoT RL training engineerable.

The ByteDance + Tsinghua Collaboration Model

The repo name BytedTsinghua-SIA says it all: ByteDance provides compute and engineering teams; Tsinghua AIR contributes the algorithmic ideas.

The joint-lab pattern between Chinese tech giants and top universities has been running since USTC-Microsoft and THU-IBM days. What's distinctive about DAPO is that ByteDance is open-sourcing industrial-grade RL training infrastructure directly — data pipelines, distributed rollout, reward services.

Behind this is a strategic bet. In the RL infra space, the ByteDance ecosystem is playing a "public utility" role akin to a Linux Foundation: get everyone building on your substrate, and the moat becomes data and application scenarios.

What the Code Tells Us

After cloning and poking around:

DAPO/
├── README.md
├── verl/                 # Training framework, modified from ByteDance's verl
├── examples/             # Config examples
│   └── aime_mix_train.yaml
├── scripts/              # Launch scripts
├── reward_server/        # Reward service (deployed separately)
└── docs/                 # Algorithm notes

A few observations:

1. The training framework is verl-based

The verl/ directory confirms DAPO is built on ByteDance's own verl (Volcano Engine Reinforcement Learning), which sits on Ray + vLLM and supports PPO, GRPO, and other algorithms.

DAPO adds two layers of customization on top:

  • Algorithm layer: Decoupled Clip and Dynamic Sampling.
  • System layer: decoupled reward service and rollout optimizations.

2. The data format is production-grade

Schema from examples/aime_mix_train.yaml:

data:
  train_files: ~/data/rl/aime_mix/train.parquet
  val_files: ~/data/rl/aime_mix/test.parquet
  prompt_key: prompt
  max_prompt_length: 2048
  max_response_length: 8192
  train_batch_size: 512
  val_batch_size: 256

Notice max_response_length: 8192 — that's a textbook long-CoT setting. Standard RLHF tasks cap out around 2048.

3. Can a regular researcher actually run this?

Bar is high:

  • At minimum 4-8 H100/A100 GPUs (the README suggests 16).
  • Tens of GB of math problem data (AIME, MATH, NuminaMath).
  • Ray cluster deployment know-how.

But the repo ships complete scripts/, and if you have the hardware, bash scripts/run_dapo.sh should theoretically get you off the ground. Compared to "open-sourced but unreproducible" releases, DAPO is on the generous side.

Who Should Care

Prioritized:

  1. Small reasoning-model teams: Qwen, DeepSeek, and Yi are already running long-CoT RL, but smaller teams lack the engineering muscle. DAPO gives them a relatively complete starting point.
  2. Anyone burned by GRPO/PPO: DAPO's dynamic sampling and decoupled clip are directly portable engineering lessons.
  3. RL infra investors and watchers: The ByteDance ecosystem (Volcano Engine + Seed) keeps doubling down on RL infrastructure — a signal for whether RL infra is forming into a standalone category.
  4. Academic researchers: With Tsinghua AIR backing, full code, and the paper, this is a textbook industrial-grade baseline.

Editorial Notes

ByteDance's open-source cadence around RL infra follows a clear logic: free infrastructure; the real moat is data and scenarios.

This differs from Meta's Llama playbook. Meta open-sourced model weights to counter OpenAI's API lock-in — the business case being "more people use my models → more people buy my GPUs/ads." ByteDance's DAPO, verl, and UI-TARS releases are closer to: make RL training accessible to everyone → RL becomes a utility → future revenue comes from compute and APIs.

What's specifically new this time? Three details:

  1. Engineering long-CoT RL used to be a closed-shop problem only OpenAI and Anthropic could solve. DAPO turns it into an open problem.
  2. Decoupled Clip + Dynamic Sampling isn't an algorithmic innovation — it's an engineering one. That tells you RL training's bottleneck is no longer algorithms but data sampling efficiency and GPU utilization.
  3. The repo naming BytedTsinghua-SIA reflects a new industry-academia collaboration model — the giant provides compute infrastructure, the university supplies algorithmic ideas, and both sign off but own their respective strengths.

Open questions: I haven't reproduced DAPO's training cost or actual convergence speed yet. The README numbers might be optimistic. If you've run it, drop your real-world experience in the comments.

References:

Source: https://github.com/BytedTsinghua-SIA/DAPO