Blog

Building Support for On-Policy Self-Distillation in SkyRL

In collaboration with UC Berkeley SkyLab

Liheng L.Dev P.UC Berkeley SkyLabMichael E.Eric T.UC Berkeley SkyLab

Introduction

A cheat sheet can help you work through a problem you would otherwise get stuck on, and with practice you may learn to solve similar problems without it. Self-distillation applies a similar idea using two instances of the same model. One attempts the problem on its own, while the other, called the “teacher,” receives a cheat sheet in the form of a solution or feedback. We use the teacher’s predictions to train the first instance to answer as if it had the cheat sheet, without giving it that extra context.

We built support for Self-Distillation Policy Optimization (SDPO) in SkyRL, an open-source framework for training language models with reinforcement learning, and used it to study learning on math problems, questions about PDF documents, and tasks that ask a model to recreate chart images in code. The implementation is in this SkyRL pull request.

With one sampled answer per prompt and a reward from the verifier, Qwen3-8B trained with SDPO reached 91.2% on MATH-500 after 6,400 responses, compared with 91.8% after 51,200 responses in our GRPO run.

Prior work. Trajectory’s Scaling SDPO demonstrated learning from one response per task, including failed attempts. We use its adaptive clipping approach in our experiments.

1. Self-distillation framework

We call the model answering without extra context the student. For each answer it generates, the teacher uses the prompt, its extra context, and the preceding tokens to predict what could come next. We update the student’s weights to bring its predictions closer to the teacher’s at each token. These answers, or rollouts, come from the current student, which makes the training on-policy.

Our SkyRL implementation supports three methods: SDPO, On-Policy Self-Distillation (OPSD), and Sample-Routed Policy Optimization (SRPO). They differ in what the teacher sees and how they combine distillation with reward-based training. All three run on FSDP and Megatron training workers.

SDPO / OPSDG rollouts per promptSDPO / OPSD1 rollout per promptSRPOG rollouts, routedPromptRolloutsTeacher readsLossxπθSDPOxysibling ✓ or feedbackOPSDxyreference solutionKL to teacher on every rollout(weight differs per token)xπθSDPOxyfeedback on yOPSDxyreference solutionKL to teacher on the rollout(no siblings needed)xπθ→ SDPOxysibling ✓→ GRPOno teacherKL to teachervaries per tokenGRPO advantageone per rollout
Figure 1.1: Teacher inputs and training objectives. In OPSD, the teacher receives a reference solution or program for each student response. SDPO uses feedback on the response or a successful answer sampled for the same prompt, called a sibling. With one response (G=1), there is no sibling to use as context, so SDPO relies on feedback. OPSD still has the reference solution. SRPO routes failed responses with a correct sibling to SDPO and all others to GRPO. GRPO compares rewards within each group and assigns each response a relative score, or advantage, that applies to all its tokens.

We can keep the teacher’s weights fixed or let them follow the student during training. A frozen teacher retains the starting weights; an exponential moving average (EMA) teacher gradually incorporates student updates. We can also use the current student directly, blocking gradients through the teacher computation. In all three cases, new context from a verifier or judge can change the teacher’s predictions.

Training objective and implementation

We use reverse KL divergence to measure the difference between the two models’ next-token probabilities. At position tt, these are pt(v)p_t(v) for the student and qt(v)q_t(v) for the teacher, over possible tokens vv. The mask MtM_t selects which positions contribute to the loss.

We can estimate this loss using only the probabilities of the token the student sampled. In our checks, however, that estimate produced a noisy gradient that often pointed in a different direction from the full-vocabulary gradient. Most runs therefore compare probabilities for the student’s 100 most likely tokens, with each model’s remaining probability mass collected in one extra bucket:

KLttop-k=∑v∈Tkpt(v) log⁡pt(v)qt(v)  +  pt(Tˉk) log⁡pt(Tˉk)qt(Tˉk),pt(Tˉk)=1−∑v∈Tkpt(v).\begin{aligned} \mathrm{KL}^{\text{top-}k}_t &= \sum_{v \in T_k} p_t(v)\,\log\frac{p_t(v)}{q_t(v)} \;+\; p_t(\bar T_k)\,\log\frac{p_t(\bar T_k)}{q_t(\bar T_k)}, \\[4pt] p_t(\bar T_k) &= 1 - \sum_{v\in T_k} p_t(v). \end{aligned}

In two matched OPSD runs on Qwen3-VL-8B with fixed teachers, this top-100-plus-tail approximation reached a comparable score in about half as many updates as the sampled-token estimate, then finished with a higher score.

586266707404008001.2k1.6kTraining updateChartMimic-600 Overall
  • Top-100 + tail
  • Sampled token only
  • Base model: 63.3
Figure 1.2: Top-100 and sampled-token KL estimators. ChartMimic-600 Overall score for OPSD at G=1 with a fixed teacher that sees the reference code. Bold curves average eight evaluations; faint curves show individual evaluations.

2. Math reasoning

Generating several responses per prompt is a major sampling cost in group-based training: GRPO needs a group to compare rewards, and sibling SDPO needs one to find a successful answer for the teacher. Trajectory’s Scaling SDPO showed that SDPO can learn from one response per task while retaining failed attempts, so we tested this setting on math problems by giving the teacher each response and feedback derived from its Math-Verify reward. We trained on correct, incorrect, and unverifiable responses, with no reference solution or sibling answer supplied to the teacher. We compared this approach with SDPO using successful siblings, GRPO, and SRPO.

All four math runs used Qwen3-8B in thinking mode and received OpenR1 prompts in the same order, with 64 prompts per update. We evaluated MATH-500 accuracy every 25 updates, sampling one answer per problem at temperature 0.6. SDPO with verifier feedback used a sampled-token loss and an importance-ratio cap of 2. Sibling SDPO used eight responses per prompt, top-100-plus-tail KL, and an EMA teacher at rate 0.05, with responses limited to 8,192 tokens.

70%80%90%100%1k2k5k10k20k50kCumulative student rollouts (log scale)MATH-500 accuracy (%)87.2%1.6k88.8%3.2k90.8%4.8k91.2%6.4k94.0%87.0%91.8%12.8k90.0%91.4%91.6%25.6k88.2%91.0%92.2%38.4k77.4%91.8%92.6%51.2k
  • SDPO G=1, verifier feedback
  • SDPO G=8, sibling solution
  • GRPO G=8
  • SRPO G=8
  • Base model: 85.6
Figure 2.1: MATH-500 accuracy by student responses. The x-axis counts cumulative student responses on a log scale. With 64 prompts per update, G=1 generates 64 responses and G=8 generates 512. Runs use one seed each.

Feedback supplied to the teacher

Correct
“This is a good response. The final answer is correct.”
Incorrect or unverifiable
“This is a bad response. The final answer is incorrect or could not be verified. Solve the original problem correctly and put the final answer in \boxed{...}.”
Show Table 2.1
Table 2.1: MATH-500 checkpoints and recorded training time
MethodGEval updateResponses to evalAccuracy (%)Logged time (h)
Base model (Qwen3-8B)—0085.6—
SDPO G=1, verifier feedback11006,40091.23.7
SDPO G=8, sibling solution810051,20077.418.4
GRPO G=8810051,20091.819.7
SRPO G=8810051,20092.617.1

Times cover the full recorded runs, not time to the reported score.

30%40%50%60%70%80%90%100%0255075Training updateTraining reward (% correct)
  • SDPO G=8, sibling solution
  • SDPO G=1, verifier feedback
  • GRPO G=8
  • SRPO G=8
Figure 2.2: Training reward, response length, and evaluation accuracy. Training reward is the fraction of responses verified as correct. Reward is shown through update 75 and response length through update 100; bold lines use a centered five-update average. The accuracy tab shows MATH-500 evaluations through update 75. Under sibling SDPO, responses shorten sharply as accuracy falls.

At the reported evaluation checkpoints, SDPO with verifier feedback reached a similar MATH-500 score to GRPO with one-eighth as many responses, generating one-eighth as many responses per update, while SRPO reached 92.6% at update 100. The savings showed up in wall-clock time too: the full recorded runs took 3.7 hours for SDPO over 100 updates and 19.7 hours for GRPO over 99 updates. SDPO with verifier feedback also hit the response length limit less often, with a truncation rate about 5% lower than GRPO’s.

3. Multimodal experiments

3.1 Document reasoning

For math, a verifier can reduce feedback to “correct” or “incorrect,” but a document answer can be partly right. We first moved beyond math by using SDPO to train Qwen3.5-9B (non-reasoning) to answer questions about PDF documents in GDP-PDF. After 30 updates, the mean rubric score rose from 28.4% to 43.9% on 40 held-out tasks and from 34.3% to 40.2% on the 100-task public set. We sampled one response per prompt and kept the teacher’s weights fixed. It received the PDF pages, question, and a judge’s feedback on each rubric criterion, but no expert answer; GPT-5.4 mini served as the judge. Answers that met every rubric criterion remained uncommon.

Figure 3.1: GDP-PDF rubric scores. Mean rubric score over three responses per task. Table 3.1 also reports pass@3: the share of tasks with at least one response that satisfies every criterion.
Show Table 3.1
Table 3.1: GDP-PDF: base model versus SDPO after 30 updates
DatasetBase pass@3SDPO pass@3Mean rubric score, base → SDPO
Held-out (40 tasks)0% (0/40)2.5% (1/40)28.4% → 43.9%
Public (100 tasks)2% (2/100)4% (4/100)34.3% → 40.2%

3.2 Chart-to-code

Chart-to-code adds another step on top of document reasoning: the model must turn what it sees into a program that reproduces the chart, and a program can run while still drawing the wrong chart type, labels, or layout. We trained Qwen3-VL-8B-Instruct to write code that reproduces a given chart image. We used VinciCoder’s Chart2Code split for training and 600 ChartMimic Direct Mimic tasks for evaluation. The Overall score averages a score from tracing the code and a GPT-4o judgment; programs that fail to run score zero.

3.2.1 Reference-guided distillation (OPSD)

We first compared two ways to learn from reference programs. Supervised fine-tuning (SFT) trains the model to reproduce them directly, while OPSD gives them only to the teacher. We ran SFT and two OPSD variants, at G=1 and G=8, from the same base model on the same 10,000 training tasks. Both OPSD variants improved while SFT showed little sustained gain. The two OPSD learning curves were similar even though the smaller group generated one-eighth as many responses per update. Across two Visual-SDPO runs, G=8 peaked at 70.9 while G=4 reached 69.5 and was roughly on par despite having a smaller group size.

5458626670050100150200250300Training stepChartMimic-600 Overall63.1763.0362.5064.1962.29064.5163.9264.2762.8756.722565.9965.5963.1362.8061.455065.0966.3064.9865.4360.777566.3866.3165.4065.6559.1210066.7566.4566.0266.6963.2512567.4468.2767.8767.9259.0815066.9068.2166.6865.8961.0117568.6567.8065.2667.3262.0520067.4369.0966.5266.7760.9222568.4669.8466.2467.7655.8125069.0370.9267.0367.1460.5627569.4869.9463.89300
  • Visual-SDPO, G=4 (256 rollouts per step)
  • Visual-SDPO, G=8 (256 rollouts per step, 32 prompts)
  • OPSD, G=1 (64 rollouts per step)
  • OPSD, G=8 (512 rollouts per step)
  • SFT on the same data
  • Zero-shot base model: 63.2
Figure 3.2: ChartMimic scores from the base model. One seed per run; scores start between 62.3 and 64.2 for the same base checkpoint. Curves are shown through update 300. Both OPSD jobs crashed at update 299, leaving update 275 as their last evaluation. The Visual-SDPO G=8 run continued to update 350, where it scored 68.6. The Entropy tab shows the four distillation runs; SFT did not log entropy.

3.2.2 Feedback-guided distillation (Visual-SDPO)

We next adapted Visual-SDPO (Dong, 2026), which swaps the reference program for feedback on the student’s own chart. A detector combines code-trace comparisons with a focused vision-model check for text, chart-type, color, and layout defects. It turns these findings into a defect list: short instructions that each name a chart element and the correction to make, such as “set the legend to ‘Walk’.” The reference program helps the detector identify defects but is never shown to the teacher. The teacher’s weights stayed fixed, and it received the rendered chart, defect list, and a better sibling’s code when available. During training, we give extra weight to code tokens linked to the detected defects and add a small GRPO loss.

Visual-SDPO loss function

The renderer links chart elements to the lines of code that produced them. For linked tokens, we assign a weight ctc_t based on the severity sts_t of the detected defect. For layout problems, severity is assessed using region overlap:

ct=1+(α−1) st,α=5.c_t = 1 + (\alpha - 1)\, s_t, \qquad \alpha = 5.

Unlinked tokens have ct=1c_t=1. We multiply each token’s loss ℓi,t\ell_{i,t} by its defect weight and entropy weight wi,tw_{i,t}, then add a small GRPO term:

ℒ=1N∑i∑tMi,t ci,t wi,t ℓi,t∑tMi,t  +  0.1 ℒGRPO.\mathcal{L} = \frac{1}{N}\sum_i \frac{\sum_t M_{i,t}\, c_{i,t}\, w_{i,t}\, \ell_{i,t}}{\sum_t M_{i,t}} \;+\; 0.1\,\mathcal{L}_{\text{GRPO}}.

We first average the weighted losses over the tokens selected by Mi,tM_{i,t} in each response, then average across the NN responses in the batch. This prevents longer responses from receiving more weight simply because they contain more tokens.

  • extra teacher context
  • defect and token weight
  • another sampled response
1 · Run the programprogram yrolloutsandboxruns yreturns tracerender+ element trace2 · DiagnoseHybrid detectorreference code never enters a promptcode-trace diff + narrow VLM passtext · type · colour · layoutdefect listfix + source linesibling pickbetter sibling’s codecode mapperc = 1 + 4 · severity3 · Teacher inputxrenderdefectssibling codeteacher-only context c(y)the student reads x onlyc per tokenweights each token’s loss
Figure 3.3: Visual feedback and token weighting. The program runs in a sandbox. A detector compares its output with the reference and produces fixes tied to source lines. The teacher receives the render, fixes, and a higher-scoring sibling’s code when available. The reference program stays inside the detector.

At update 275, Visual-SDPO scored 69.0, compared with 67.0 for OPSD at G=1 and 67.1 at G=8. The Visual-SDPO run started from the base model, used G=4 and top-100-plus-tail KL, and reached 69.5 by update 300. Despite its smaller group, the G=4 run roughly matched G=8, staying within two points at every evaluation through update 300 (69.5 versus 69.9 at update 300) while generating half as many responses per prompt.

Teacher stability

Getting here depended on how the teacher was updated. The first clue came from an early OPSD run that improved while its teacher was accidentally left frozen, then declined after a restart made the teacher track the student. To test whether keeping the teacher fixed would help, we started four runs varying whether the teacher stayed frozen or used EMA updates and whether it received the student’s rendered chart.

3040506070800100200300400500600700775Training updateChartMimic-600 Overall74.7974.6974.6875.45075.3275.4074.5575.182576.1476.5075.7174.935076.7675.9675.0874.367575.4675.3775.5074.6310075.0975.5975.9774.1112575.0375.6976.4175.3515075.2274.7075.4675.2717575.8374.4775.1675.8020075.7674.9775.1375.0622576.0073.5073.4876.2025076.7673.5572.6975.8227575.5773.4772.5176.6230076.7572.7671.9875.5332576.7072.2171.3077.0335076.7071.0971.4876.7037576.1168.3970.1876.7040075.4164.3670.0976.2342575.3353.9770.4076.4945076.0054.6570.2576.7547575.8152.6571.3676.4250076.0845.5470.3376.6052576.1841.5368.7576.4455075.9741.9869.6275.4857575.7441.2267.4574.6760074.6737.2768.2075.2862576.5667.1473.8565076.9768.1874.5867576.6066.8676.4470075.1465.9175.6472574.3764.3475.6275077.1576.59775
  • Frozen teacher, render in context
  • EMA teacher, render in context
  • EMA teacher, render removed
  • Frozen teacher, render removed
  • SFT start: 74.1
Figure 3.4: Teacher updates and render context. Four runs from the same SFT checkpoint compare frozen teachers with EMA teachers (rate 0.05), each with and without the student’s rendered chart. The Teacher KL tab shows student–teacher divergence averaged over ten updates.

The model’s score fell in both runs with an EMA teacher, with a faster decline when the teacher also saw the rendered chart. Runs with frozen teachers remained stable, which is why the OPSD and Visual-SDPO runs in Figure 3.2 use frozen teachers.

Feedback evaluation

The rendered chart also changed how the teacher judged the student. In a separate probe, it accepted 74% of defective responses when shown the rendered chart, compared with 1% or less when given a sibling’s code or the reference program. To find which context actually helped it produce better solutions, we asked the model to rewrite the student’s solutions for all 600 charts using different combinations of feedback, images, and code, then scored the results.

  • Shows a better program
  • Feedback, images, or the student’s own code
Feedback only, no codeaverage of 7 feedback-only formats
−1.2
Student’s own code + feedbackaverage of 9 variants
+2.0
Student’s own code + side-by-side image
−6.3
A better sibling’s chart image
+0.4
A better sibling’s codeaverage of 2 variants
+9.9
Feedback + own code + sibling codeaverage of 2 variants
+11.7
The reference program
+19.3

Change in Overall versus the student’s own answer (73.82)

Figure 3.5: Effect of feedback context on chart rewrites. Change in ChartMimic-600 Overall score after the frozen SFT model rewrites the student’s answer. The starting answers score 73.82 on average. The table includes 95% intervals for each change.
Show data table
ContextVariantΔ Overall95% interval
Feedback only, no codeF1 rubric JSON−0.96[−3.03, +1.03]
F2 render or error−1.00[−2.85, +0.79]
F3 VLM defect table+0.04[−1.89, +1.93]
F4 hybrid, describe−2.70[−4.73, −0.75]
F5 hybrid, directive−1.89[−4.10, +0.27]
Paper option 1−1.21[−3.07, +0.62]
Paper option 2−0.95[−2.89, +0.86]
Own code + feedbackOwn code only+0.49[−0.19, +1.24]
+ F1+1.27[+0.28, +2.38]
+ F2+2.26[+1.03, +3.60]
+ F3+1.44[+0.47, +2.49]
+ F4+2.07[+0.74, +3.51]
+ F5+2.46[+1.09, +3.91]
+ paper option 1+2.27[+1.03, +3.61]
+ paper option 2+2.34[+1.11, +3.64]
+ F5, image removed+3.05[+1.65, +4.51]
Own code + side-by-side image−6.29[−8.16, −4.35]
Sibling chart only+0.43[−0.17, +1.13]
Sibling codeCode+9.76[+8.09, +11.55]
Code + chart+10.02[+8.39, +11.74]
Feedback + own code + sibling codeAnswer directly+11.50[+9.57, +13.51]
Self-review first+11.98[+9.89, +14.16]
Reference program+19.28[+16.43, +22.01]

With feedback alone, the teacher’s rewrites did not reliably improve on the original answers. Several feedback formats helped when the student’s code was included. Rewrites using the code and the defect list scored 2.46 points above the original answers, or 3.05 points when the image was omitted. The gains were larger with a better solution: 9.76 points with a better sibling’s code and 19.28 with the reference program.

Reference chart

Reference: nine overlapping filled density curves coloured from blue to red

Student’s answer40.8

Student render: a single contour map instead of density curves

Defect list given to the teacher

- [chart_type] Add 9 fill_between calls in subplot 1.
- [chart_type] Remove 1 contourf call in subplot 1.
- [visual] Contour plot fills entire area, unlike reference.
- [text] Remove the x-axis label in subplot 1 "Time"; the reference has no such text.
Figure 3.6: Example defect feedback. For task density_1, the teacher receives the student’s rendered chart and the defect list. Each instruction identifies a chart element and a correction.

In Figure 3.7, the trained model corrects a chart-type error, and the score rises from 63.8 to 99.6. In another example, a program that failed with a syntax error now runs and scores 96.0. The overall gain still includes some regressions: a radar chart the base model already drew well falls from 99.6 to 89.8.

errorpoint_11

Reference

Reference: points with error bars and a dashed average line

Base model63.8

Base model: yellow bars with error bars instead of points

Trained model99.6

Trained model: points with error bars matching the reference
Figure 3.7: Student outputs before and after training. Selected outputs from the base and trained models. Each response uses greedy decoding, which chooses the most likely token at each step. Scores are ChartMimic Overall.

4. Next steps

Next, we would repeat the chart comparisons across more runs and vary sibling code and token weighting separately. We can also use the rewrite probe to choose feedback formats for training, then measure whether improvements in the teacher’s answers carry through to the student.

If you want to learn more about our research work or off-the-shelf data, please reach out to research@afterquery.com.

We’re working on hard problems where post-training algorithms meet expert data at scale. If those sound like the kinds of problems you’re interested in, join us.

AfterQuery is an applied research lab curating data solutions to accelerate foundation model development.

References

Cite this post

BibTeX
@misc{afterquery2026selfdistillation,
  author = {Lai, Liheng and Patel, Dev and Evans, Michael and Ta, Eric},
  title = {{Building Support for On-Policy Self-Distillation in SkyRL}},
  year = {2026},
  month = oct,
  howpublished = {AfterQuery}
}

Related articles