
Building Support for On-Policy Self-Distillation in SkyRL
In collaboration with UC Berkeley SkyLab
Liheng L.
Dev P.UC Berkeley SkyLab
Michael E.
Eric T.UC Berkeley SkyLabIntroduction
A cheat sheet can help you work through a problem you would otherwise get stuck on, and with practice you may learn to solve similar problems without it. Self-distillation applies a similar idea using two instances of the same model. One attempts the problem on its own, while the other, called the “teacher,” receives a cheat sheet in the form of a solution or feedback. We use the teacher’s predictions to train the first instance to answer as if it had the cheat sheet, without giving it that extra context.
We built support for Self-Distillation Policy Optimization (SDPO) in SkyRL, an open-source framework for training language models with reinforcement learning, and used it to study learning on math problems, questions about PDF documents, and tasks that ask a model to recreate chart images in code. The implementation is in this SkyRL pull request.
With one sampled answer per prompt and a reward from the verifier, Qwen3-8B trained with SDPO reached 91.2% on MATH-500 after 6,400 responses, compared with 91.8% after 51,200 responses in our GRPO run.
Prior work. Trajectory’s Scaling SDPO demonstrated learning from one response per task, including failed attempts. We use its adaptive clipping approach in our experiments.
1. Self-distillation framework
We call the model answering without extra context the student. For each answer it generates, the teacher uses the prompt, its extra context, and the preceding tokens to predict what could come next. We update the student’s weights to bring its predictions closer to the teacher’s at each token. These answers, or rollouts, come from the current student, which makes the training on-policy.
Our SkyRL implementation supports three methods: SDPO, On-Policy Self-Distillation (OPSD), and Sample-Routed Policy Optimization (SRPO). They differ in what the teacher sees and how they combine distillation with reward-based training. All three run on FSDP and Megatron training workers.
We can keep the teacher’s weights fixed or let them follow the student during training. A frozen teacher retains the starting weights; an exponential moving average (EMA) teacher gradually incorporates student updates. We can also use the current student directly, blocking gradients through the teacher computation. In all three cases, new context from a verifier or judge can change the teacher’s predictions.
Training objective and implementation
We use reverse KL divergence to measure the difference between the two models’ next-token probabilities. At position , these are for the student and for the teacher, over possible tokens . The mask selects which positions contribute to the loss.
We can estimate this loss using only the probabilities of the token the student sampled. In our checks, however, that estimate produced a noisy gradient that often pointed in a different direction from the full-vocabulary gradient. Most runs therefore compare probabilities for the student’s 100 most likely tokens, with each model’s remaining probability mass collected in one extra bucket:
In two matched OPSD runs on Qwen3-VL-8B with fixed teachers, this top-100-plus-tail approximation reached a comparable score in about half as many updates as the sampled-token estimate, then finished with a higher score.
- Top-100 + tail
- Sampled token only
- Base model: 63.3
Large disagreements between the two models can dominate an update. To limit their effect, we clip the log-probability difference at a threshold . The first update is unclipped. After that, the threshold is times the average magnitude of these differences over earlier updates:
We use this adaptive clipping throughout our experiments, following Trajectory’s Scaling SDPO, and apply a separate importance-ratio cap to the sampled-token loss. To test its effect, we compared two math runs in which sampling lagged training by one optimizer step. Without this clipping, accuracy fell by about ten percentage points after update 50; clipping avoided most of that decline.
- Advantage clip at 3×
- No advantage clip
- Advantage clip at 3×
- No advantage clip
Following Li et al. (2026), we give more weight to positions where the teacher favors fewer next-token choices. We measure how spread out its predictions are using entropy, . When , lower entropy produces a larger weight through ; setting gives uniform weights:
We normalize these weights over the full optimizer batch, so splitting it into smaller batches leaves the objective unchanged. The SRPO paper reports gains from this weighting across five benchmarks.
2. Math reasoning
Generating several responses per prompt is a major sampling cost in group-based training: GRPO needs a group to compare rewards, and sibling SDPO needs one to find a successful answer for the teacher. Trajectory’s Scaling SDPO showed that SDPO can learn from one response per task while retaining failed attempts, so we tested this setting on math problems by giving the teacher each response and feedback derived from its Math-Verify reward. We trained on correct, incorrect, and unverifiable responses, with no reference solution or sibling answer supplied to the teacher. We compared this approach with SDPO using successful siblings, GRPO, and SRPO.
All four math runs used Qwen3-8B in thinking mode and received OpenR1 prompts in the same order, with 64 prompts per update. We evaluated MATH-500 accuracy every 25 updates, sampling one answer per problem at temperature 0.6. SDPO with verifier feedback used a sampled-token loss and an importance-ratio cap of 2. Sibling SDPO used eight responses per prompt, top-100-plus-tail KL, and an EMA teacher at rate 0.05, with responses limited to 8,192 tokens.
- SDPO G=1, verifier feedback
- SDPO G=8, sibling solution
- GRPO G=8
- SRPO G=8
- Base model: 85.6
Feedback supplied to the teacher
- Correct
- “This is a good response. The final answer is correct.”
- Incorrect or unverifiable
- “This is a bad response. The final answer is incorrect or could not be verified. Solve the original problem correctly and put the final answer in \boxed{...}.”
Show Table 2.1
| Method | G | Eval update | Responses to eval | Accuracy (%) | Logged time (h) |
|---|---|---|---|---|---|
| Base model (Qwen3-8B) | — | 0 | 0 | 85.6 | — |
| SDPO G=1, verifier feedback | 1 | 100 | 6,400 | 91.2 | 3.7 |
| SDPO G=8, sibling solution | 8 | 100 | 51,200 | 77.4 | 18.4 |
| GRPO G=8 | 8 | 100 | 51,200 | 91.8 | 19.7 |
| SRPO G=8 | 8 | 100 | 51,200 | 92.6 | 17.1 |
Times cover the full recorded runs, not time to the reported score.
- SDPO G=8, sibling solution
- SDPO G=1, verifier feedback
- GRPO G=8
- SRPO G=8
- SDPO G=8, sibling solution
- SDPO G=1, verifier feedback
- GRPO G=8
- SRPO G=8
- SDPO G=8, sibling solution
- SDPO G=1, verifier feedback
- GRPO G=8
- SRPO G=8
At the reported evaluation checkpoints, SDPO with verifier feedback reached a similar MATH-500 score to GRPO with one-eighth as many responses, generating one-eighth as many responses per update, while SRPO reached 92.6% at update 100. The savings showed up in wall-clock time too: the full recorded runs took 3.7 hours for SDPO over 100 updates and 19.7 hours for GRPO over 99 updates. SDPO with verifier feedback also hit the response length limit less often, with a truncation rate about 5% lower than GRPO’s.
3. Multimodal experiments
3.1 Document reasoning
For math, a verifier can reduce feedback to “correct” or “incorrect,” but a document answer can be partly right. We first moved beyond math by using SDPO to train Qwen3.5-9B (non-reasoning) to answer questions about PDF documents in GDP-PDF. After 30 updates, the mean rubric score rose from 28.4% to 43.9% on 40 held-out tasks and from 34.3% to 40.2% on the 100-task public set. We sampled one response per prompt and kept the teacher’s weights fixed. It received the PDF pages, question, and a judge’s feedback on each rubric criterion, but no expert answer; GPT-5.4 mini served as the judge. Answers that met every rubric criterion remained uncommon.
- Base model
- SDPO, update 30
Show Table 3.1
| Dataset | Base pass@3 | SDPO pass@3 | Mean rubric score, base → SDPO |
|---|---|---|---|
| Held-out (40 tasks) | 0% (0/40) | 2.5% (1/40) | 28.4% → 43.9% |
| Public (100 tasks) | 2% (2/100) | 4% (4/100) | 34.3% → 40.2% |
3.2 Chart-to-code
Chart-to-code adds another step on top of document reasoning: the model must turn what it sees into a program that reproduces the chart, and a program can run while still drawing the wrong chart type, labels, or layout. We trained Qwen3-VL-8B-Instruct to write code that reproduces a given chart image. We used VinciCoder’s Chart2Code split for training and 600 ChartMimic Direct Mimic tasks for evaluation. The Overall score averages a score from tracing the code and a GPT-4o judgment; programs that fail to run score zero.
3.2.1 Reference-guided distillation (OPSD)
We first compared two ways to learn from reference programs. Supervised fine-tuning (SFT) trains the model to reproduce them directly, while OPSD gives them only to the teacher. We ran SFT and two OPSD variants, at G=1 and G=8, from the same base model on the same 10,000 training tasks. Both OPSD variants improved while SFT showed little sustained gain. The two OPSD learning curves were similar even though the smaller group generated one-eighth as many responses per update. Across two Visual-SDPO runs, G=8 peaked at 70.9 while G=4 reached 69.5 and was roughly on par despite having a smaller group size.
- Visual-SDPO, G=4 (256 rollouts per step)
- Visual-SDPO, G=8 (256 rollouts per step, 32 prompts)
- OPSD, G=1 (64 rollouts per step)
- OPSD, G=8 (512 rollouts per step)
- SFT on the same data
- Zero-shot base model: 63.2
- Visual-SDPO, G=4 (256 rollouts per step)
- Visual-SDPO, G=8 (256 rollouts per step, 32 prompts)
- OPSD, G=1 (64 rollouts per step)
- OPSD, G=8 (512 rollouts per step)
3.2.2 Feedback-guided distillation (Visual-SDPO)
We next adapted Visual-SDPO (Dong, 2026), which swaps the reference program for feedback on the student’s own chart. A detector combines code-trace comparisons with a focused vision-model check for text, chart-type, color, and layout defects. It turns these findings into a defect list: short instructions that each name a chart element and the correction to make, such as “set the legend to ‘Walk’.” The reference program helps the detector identify defects but is never shown to the teacher. The teacher’s weights stayed fixed, and it received the rendered chart, defect list, and a better sibling’s code when available. During training, we give extra weight to code tokens linked to the detected defects and add a small GRPO loss.
Visual-SDPO loss function
The renderer links chart elements to the lines of code that produced them. For linked tokens, we assign a weight based on the severity of the detected defect. For layout problems, severity is assessed using region overlap:
Unlinked tokens have . We multiply each token’s loss by its defect weight and entropy weight , then add a small GRPO term:
We first average the weighted losses over the tokens selected by in each response, then average across the responses in the batch. This prevents longer responses from receiving more weight simply because they contain more tokens.
- extra teacher context
- defect and token weight
- another sampled response
At update 275, Visual-SDPO scored 69.0, compared with 67.0 for OPSD at G=1 and 67.1 at G=8. The Visual-SDPO run started from the base model, used G=4 and top-100-plus-tail KL, and reached 69.5 by update 300. Despite its smaller group, the G=4 run roughly matched G=8, staying within two points at every evaluation through update 300 (69.5 versus 69.9 at update 300) while generating half as many responses per prompt.
Teacher stability
Getting here depended on how the teacher was updated. The first clue came from an early OPSD run that improved while its teacher was accidentally left frozen, then declined after a restart made the teacher track the student. To test whether keeping the teacher fixed would help, we started four runs varying whether the teacher stayed frozen or used EMA updates and whether it received the student’s rendered chart.
- Frozen teacher, render in context
- EMA teacher, render in context
- EMA teacher, render removed
- Frozen teacher, render removed
- SFT start: 74.1
- Frozen teacher, render in context
- EMA teacher, render in context
- EMA teacher, render removed
- Frozen teacher, render removed
The model’s score fell in both runs with an EMA teacher, with a faster decline when the teacher also saw the rendered chart. Runs with frozen teachers remained stable, which is why the OPSD and Visual-SDPO runs in Figure 3.2 use frozen teachers.
Feedback evaluation
The rendered chart also changed how the teacher judged the student. In a separate probe, it accepted 74% of defective responses when shown the rendered chart, compared with 1% or less when given a sibling’s code or the reference program. To find which context actually helped it produce better solutions, we asked the model to rewrite the student’s solutions for all 600 charts using different combinations of feedback, images, and code, then scored the results.
- Shows a better program
- Feedback, images, or the student’s own code
Change in Overall versus the student’s own answer (73.82)
Show data table
| Context | Variant | Δ Overall | 95% interval |
|---|---|---|---|
| Feedback only, no code | F1 rubric JSON | −0.96 | [−3.03, +1.03] |
| F2 render or error | −1.00 | [−2.85, +0.79] | |
| F3 VLM defect table | +0.04 | [−1.89, +1.93] | |
| F4 hybrid, describe | −2.70 | [−4.73, −0.75] | |
| F5 hybrid, directive | −1.89 | [−4.10, +0.27] | |
| Paper option 1 | −1.21 | [−3.07, +0.62] | |
| Paper option 2 | −0.95 | [−2.89, +0.86] | |
| Own code + feedback | Own code only | +0.49 | [−0.19, +1.24] |
| + F1 | +1.27 | [+0.28, +2.38] | |
| + F2 | +2.26 | [+1.03, +3.60] | |
| + F3 | +1.44 | [+0.47, +2.49] | |
| + F4 | +2.07 | [+0.74, +3.51] | |
| + F5 | +2.46 | [+1.09, +3.91] | |
| + paper option 1 | +2.27 | [+1.03, +3.61] | |
| + paper option 2 | +2.34 | [+1.11, +3.64] | |
| + F5, image removed | +3.05 | [+1.65, +4.51] | |
| Own code + side-by-side image | −6.29 | [−8.16, −4.35] | |
| Sibling chart only | +0.43 | [−0.17, +1.13] | |
| Sibling code | Code | +9.76 | [+8.09, +11.55] |
| Code + chart | +10.02 | [+8.39, +11.74] | |
| Feedback + own code + sibling code | Answer directly | +11.50 | [+9.57, +13.51] |
| Self-review first | +11.98 | [+9.89, +14.16] | |
| Reference program | +19.28 | [+16.43, +22.01] |
With feedback alone, the teacher’s rewrites did not reliably improve on the original answers. Several feedback formats helped when the student’s code was included. Rewrites using the code and the defect list scored 2.46 points above the original answers, or 3.05 points when the image was omitted. The gains were larger with a better solution: 9.76 points with a better sibling’s code and 19.28 with the reference program.
Reference chart

Student’s answer40.8

Defect list given to the teacher
- [chart_type] Add 9 fill_between calls in subplot 1. - [chart_type] Remove 1 contourf call in subplot 1. - [visual] Contour plot fills entire area, unlike reference. - [text] Remove the x-axis label in subplot 1 "Time"; the reference has no such text.
In Figure 3.7, the trained model corrects a chart-type error, and the score rises from 63.8 to 99.6. In another example, a program that failed with a syntax error now runs and scores 96.0. The overall gain still includes some regressions: a radar chart the base model already drew well falls from 99.6 to 89.8.
errorpoint_11
Reference

Base model63.8

Trained model99.6

line_52
Reference

Base model0.0
Did not run
SyntaxError: unterminated string literal in a subplot title
Trained model96.0

radar_20
Reference

Base model99.6

Trained model89.8

4. Next steps
Next, we would repeat the chart comparisons across more runs and vary sibling code and token weighting separately. We can also use the rewrite probe to choose feedback formats for training, then measure whether improvements in the teacher’s answers carry through to the student.
If you want to learn more about our research work or off-the-shelf data, please reach out to research@afterquery.com.
We’re working on hard problems where post-training algorithms meet expert data at scale. If those sound like the kinds of problems you’re interested in, join us.
AfterQuery is an applied research lab curating data solutions to accelerate foundation model development.
References
- Trajectory: Scaling SDPO. Neil Kale, 2026.
- Reinforcement Learning via Self-Distillation. Hübotter et al., 2026. SDPO.
- Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. Zhao et al., 2026. OPSD.
- Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing. Li et al., 2026. SRPO.
- Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts. Dong, 2026. Visual-SDPO.
Cite this post
@misc{afterquery2026selfdistillation,
author = {Lai, Liheng and Patel, Dev and Evans, Michael and Ta, Eric},
title = {{Building Support for On-Policy Self-Distillation in SkyRL}},
year = {2026},
month = oct,
howpublished = {AfterQuery}
}


