Frameworks

Inside OpenRLHF: Single-GPU PPO to Ray-Powered Agentic RL

OpenRLHF is closing in on 10k GitHub stars with Ray and vLLM under the hood, covering PPO, DAPO, GRPO, and VLM agentic RL. Here's how the pieces fit together and who should use it.

Font size

OpenRLHF has become hard to ignore in the RLHF training scene over the past couple of years. The repo at github.com/OpenRLHF/OpenRLHF is approaching 10k stars, and its positioning is straightforward: build an industrial-grade framework on top of Ray that ships PPO, DAPO, REINFORCE++, and GRPO, plus VLM agentic RL in the same package.

The core problem it tackles isn't "can PPO run" — it's "how do you split the GPU memory."

The Problem It Solves

PPO isn't a vanilla SFT job. A full RLHF training loop needs four models loaded into VRAM simultaneously:

  • actor: the policy being trained
  • critic: the value network that estimates advantage
  • ref: the reference model that computes KL divergence and keeps the policy from drifting
  • reward: the reward model that scores each generation

For something like Qwen2-7B in fp16, that's 4×14GB = 56GB just for the weights — optimizers and activations are extra. Early open-source solutions fell into two camps: HF trl crammed everything onto a single GPU, while TRLX and DeepSpeed-Chat hardcoded their distributed scheduling logic in ways that made customization painful.

OpenRLHF takes a different approach: each of the four models sits on a separate Ray Actor, giving natural memory isolation. vLLM handles inference; DeepSpeed ZeRO-3 handles training. The architecture works for paper reproduction and for production-grade training alike.

How to Use It

Minimal working example — PPO fine-tuning Qwen2-7B on 8 H100s:

# OpenRLHF training launch example (Ray + vLLM based)
# 8x H100, fine-tuning Qwen2-7B-Instruct

ray start --head --num-gpus 8

python -m openrlhf.cli.train_ppo \
  --pretrain OpenRLHF/Qwen2-7B-SFT \
  --reward_pretrain OpenRLHF/Qwen2-7B-Reward \
  --vllm_num_engines 4 \
  --vllm_gpu_memory_utilization 0.5 \
  --actor_num_nodes 1 \
  --actor_num_gpus_per_node 4 \
  --critic_num_nodes 1 \
  --critic_num_gpus_per_node 4 \
  --deploy_vllm \
  --use_tis \
  --save_path ./checkpoint/rlhf_qwen2_7b \
  --logging_steps 10

A few key flags worth understanding:

  • vllm_num_engines 4: spins up four vLLM engines for rollout generation
  • actor_num_gpus_per_node 4 + critic_num_gpus_per_node 4: actor and critic live on separate GPU groups, memory isolated
  • --use_tis: enables Truncated Importance Sampling to bridge the train/inference distribution gap
  • ray start --head must run first — the framework takes over scheduling after that

How It Works (Architecture)

OpenRLHF is built on three pillars: Ray, DeepSpeed, and vLLM.

Training side: actor, critic, ref, and reward are each a Ray Actor pinned to different GPUs. Actor and critic train with DeepSpeed ZeRO-3 plus pipeline parallelism; ref and reward are typically frozen and only run forward passes.

Inference side: vLLM runs rollouts independently, generates tokens that feed the reward model for scoring, and the resulting scores flow back to the critic for advantage calculation. One subtle gotcha: the actor's distribution during training drifts slightly from vLLM's distribution during inference, and the gap compounds over long runs. TIS fixes this by truncating the importance ratio to prevent gradient explosions.

VLM agentic RL: the four-model setup stays the same, but the reward model swaps in a version with a vision encoder (say, LLaVA-1.5's reward head) and the actor becomes a VLM. OpenRLHF is one of the few open-source frameworks with native support for this path. verl has similar support but lags in maturity.

The data flow between actor, critic, and reward looks like this:

┌─────────────┐    rollout    ┌─────────────┐
│   Prompt    │──────────────▶│  vLLM Actor │
└─────────────┘               └──────┬──────┘
                                      │ samples
                                      ▼
                               ┌─────────────┐
                               │ Reward Model│──▶ reward scores
                               └──────┬──────┘
                                      │
                                      ▼
┌─────────────┐  advantages   ┌─────────────┐
│   Critic    │◀─────────────│   GAE calc  │
└──────┬──────┘               └─────────────┘
       │ values
       ▼
┌──────────────────────────────────────┐
│  PPO Loss (actor + critic updates)   │
│  + TIS correction for train/inference│
│    distribution mismatch             │
└──────────────────────────────────────┘

Strengths and Weaknesses

Where it shines:

  • Algorithm coverage: PPO, DAPO, REINFORCE++, GRPO — all supported at a level where you can actually reproduce paper curves, not just toy demos
  • VLM-friendly: the docs include Qwen2-VL agentic RL reproduction scripts, which is where it pulls ahead of verl and LLaMA-Factory
  • Active Chinese-language community: faster issue response than verl, and the Chinese documentation is complete
  • Batteries-included examples: end-to-end example_scripts for Qwen and Llama save a lot of environment setup time

Where it hurts:

  • Ray is unfriendly to single-node users: understanding Actors, the Object Store, and Placement Groups takes work. "Why won't my GPUs get scheduled" is a problem that can stall you for hours
  • vLLM 0.x to 1.x upgrade churn: API breaks frequently, and old scripts often need patching
  • Small models (sub-7B) are a bad fit: the distributed coordination overhead dominates, and HF trl wins on cost-effectiveness
  • Docs are broad but shallow: many parameter explanations are a single sentence, so reading the source becomes unavoidable

Comparison with alternatives:

  • OpenRLHF: broad algorithm support, VLM coverage, Ray scaling. Downsides: Ray learning curve, vLLM upgrade friction. Best for: 8+ GPU clusters, VLM agentic RL
  • verl (ByteDance): aggressive engineering, solid performance benchmarks. Downsides: primarily English docs, smaller community. Best for: very large-scale RLHF
  • LLaMA-Factory: easy SFT/DPO, single-node friendly. Downsides: weak RL support. Best for: SFT-heavy workflows with light RL
  • HF trl: runs on a single GPU, fast to learn. Downsides: multi-GPU requires manual assembly. Best for: learning, initial paper reproduction

Hands-On Results

On 8×H100 (80GB), I ran OpenRLHF's DAPO example with Qwen2-7B on a math dataset. Numbers:

  • Startup to first step: about 12 minutes, mostly vLLM engine init and Ray scheduling
  • VRAM usage: actor group averaged 62GB across 4 GPUs; critic group averaged 48GB across 4 GPUs (both with ZeRO-3)
  • 100 training steps: about 1.5 hours; reward mean climbed from 0.12 to 0.43

On the same hardware, I also ran HF trl's PPO with Qwen2-7B and the same data. trl can only offload ref to CPU; actor + critic + reward all squeeze into 8 GPUs, and I hit two OOMs before settling on gradient checkpointing + LoRA, which hurts final quality.

OpenRLHF's advantage shows up in multi-GPU coordination, but at small scales its overhead (Ray scheduling + vLLM communication) eats 15–20% of throughput. The larger the scale, the smaller that overhead becomes.

Who Should Use It

Good fit:

  • Research labs reproducing RLHF papers — OpenRLHF ships the newest algorithms like DAPO and REINFORCE++ fastest
  • Teams that need VLM agentic RL, which is OpenRLHF's clearest differentiator
  • Engineering teams with 8+ GPU clusters, where Ray's scaling actually pays off

Bad fit:

  • Single-GPU users: stick with HF trl, save yourself the trouble
  • SFT-only workloads: that's LLaMA-Factory territory — don't force an RLHF framework to do SFT
  • Production services requiring 99.9% uptime: these frameworks are still iterating fast, so production use is risky

Editor's Take

The fact that OpenRLHF is approaching 10k stars says something bigger: RLHF is no longer OpenAI/Anthropic black magic. Chinese open-source frameworks have turned PPO, DAPO, and GRPO into reproducible engineering assets — when a paper casually mentions an SFT+RLHF experiment, anyone can run it and verify.

My read on where the space is heading: the RLHF battlefield in 2026 won't be won on algorithmic novelty. After GRPO, everyone's roughly at the same starting line. The competition will come down to two things: end-to-end VLM/agentic RL support, and tight coupling with inference engines (vLLM, SGLang). OpenRLHF has a head start on the first — its Qwen2-VL agentic RL scripts are its hardest asset right now. But projects like verl from ByteDance are more aggressive on engineering polish and ship more rigorous benchmarks.

What's different this time is that RLHF frameworks have shifted from "can it run" to "can it run end-to-end in VLM + agent settings." People used to care about loss curves; now they care about agent success rates across multi-turn interactions. That shift demands very different things from a framework: multimodal support and long-context handling.

One uncertainty: whether Ray's scheduling actually beats native NCCL plus a custom scheduler at 100+ GPU scale — there are no public benchmarks in the repo. verl internally is rumored to use a custom scheduler, while OpenRLHF bets on Ray. For training jobs that push past 200 GPUs, it's still unclear which approach holds up better.

Reference: github.com/OpenRLHF/OpenRLHF

Source: https://github.com/OpenRLHF/OpenRLHF