Reading a recent arXiv paper (2608.19408) that takes aim at on-policy distillation (OPD) added another layer of skepticism to how I think about post-training.
The hidden assumption behind OPD: teacher score ≈ reasoning progress
Quick refresher on OPD.
During post-training, the student model rolls out its own trajectories, and the teacher assigns each token a score as dense supervision. The approach is widespread—OpenAI's GPT line, DeepSeek's R1, certain alignment work at Anthropic—all rely on variations of this idea.
The paper's abstract puts it plainly:
pairing student-generated trajectories with dense token-level supervision from a teacher
In other words, the teacher's per-token scores serve directly as dense, token-level rewards. The whole OPD paradigm rests on an unspoken assumption: teacher-derived rewards ≈ reasoning progress.
Plain English: the tokens the teacher likes are the ones that move the student's reasoning forward.
Why the assumption falls apart
That's where the trouble lives. The authors show the assumption breaks down often enough to matter.
From the abstract:
teacher-derived rewards often conflict with genuine reasoning progress
Highly rewarded tokens aren't necessarily the ones advancing the reasoning—sometimes they're noise on a reasoning step, or even nudge the chain backward.
Not surprising, really. The teacher model is generating too, and its token-level logprobs or rewards measure more "the next most likely token" than "did this step actually push reasoning forward."
These two usually overlap. In long chain-of-thought sequences, the overlap drops sharply.
Anecdotally, when I ran the DeepSeek-R1-Distill series on a downstream task and asked the student to write a math proof, the teacher's high-scoring tokens clustered around connective phrases ("therefore," "because") and on-surface reasoning-sounding filler. The state-transition steps—the actual equation transformations—got low scores for being low-frequency expressions. Consistent with what the paper reports.
The fix: filter teacher feedback by reasoning progress
The paper's remedy is straightforward: before policy optimization, use reasoning progress as a filter to drop teacher feedback that conflicts with actual progress.
From the abstract:
filtering teacher feedback by reasoning progress before policy optimization
Where the filter sits matters a lot. It's not a loss tweak or a reward-shaping trick—it sits at the front of the OPD training loop, deciding whether a trajectory deserves teacher supervision in the first place.
My read of the loop:
# pseudo-code: reasoning-progress-aware OPD
for trajectory in student_rollouts(trajectory_buffer):
teacher_rewards = teacher.score(trajectory) # token-level
rp_scores = reasoning_progress(trajectory) # independent metric
keep_mask = agreement(teacher_rewards, rp_scores) # keep only where they agree
if keep_mask.any():
policy_optimization_step(trajectory[keep_mask], teacher_rewards[keep_mask])
Defining reasoning_progress(trajectory) is the paper's core engineering challenge—and where I'd push back most. More on that below.
What this implies: reward design in post-training is underestimated
The paper lands on a point the broader post-training community has underrated: what reward signals actually measure.
Today's competition over RLHF, DPO, OPD, and process reward models (PRM) mostly focuses on:
- how to write the loss function
- how to sample preference pairs
- how to prevent reward hacking
- how to layer in KTO / IPO / SimPO
Hardly anyone pauses to ask: is this reward signal actually measuring what I want it to measure?
OPD treats teacher token-level supervision as a stand-in for reasoning progress. That's "model outputs as ground truth" lazy thinking—and we know model outputs aren't ground truth.
The paper's filter approach is essentially admitting the gap: teacher judgment and real progress aren't the same thing, so we add an independent check.
Same logic as "retrieval isn't answer, so we re-rank" in RAG: before using model outputs as supervision, verify they're correct.
If the paper's observation holds, the ripple effect goes well past OPD. Any downstream work that uses an LLM as a scorer—reward models, judge models, automatic evaluation—needs a second look.
Open questions from my read
Four things I didn't see resolved:
1. Where does the ground truth for reasoning progress come from?
The biggest one. "Reasoning progress" sounds nice, but "did reasoning actually advance" still needs an oracle to label. For math problems, final-answer correctness works. For open-ended reasoning?
The abstract doesn't dig in. My guess: weak signals from final outcomes, or another PRM trained for the purpose—which loops back to "model supervising model."
2. Could this introduce new reward hacking?
The student optimizes on a filtered subset and could easily learn a writing style that fools the progress filter. More explicit intermediate steps, more "therefore" / "hence" connectors—none of which necessarily helps final-answer accuracy, but all of which can game a progress metric.
3. Does the filter stay stable as the training distribution drifts?
Practical concern. The student's distribution drifts during training. A filter calibrated on the base model might not hold up on late-training checkpoints. Does it need periodic recalibration? The paper doesn't say.
4. Compute overhead
Adding reasoning-progress evaluation means two passes per student rollout (forward + progress scoring). The paper doesn't mention overhead. For large-scale training, that multiplier isn't trivial.
One-line takeaway
OPD uses the teacher as judge, but the judge has its own biases. The paper's reminder: after a year of algorithmic churn in post-training, the next real bottleneck is likely the quality of reward signals—not the loss function.
Reference: arXiv 2608.19408