Improving Qwen3.5-9B on τ³-Banking using PivotRL and fewer than 250 tasks
Michael E.
Parth P.
Spencer M.We improved Qwen3.5-9B from 11.3% to 17.5% pass@5 on τ³-Banking using AfterQuery’s off-the-shelf customer service tool use data. We first distilled Qwen3.6-27B into the smaller model, then applied PivotRL using teacher solutions for only 50 tasks.
Methodology
Stage 1 — Hybrid OPD
On-policy distillation (OPD) trains a student using a teacher’s token-level feedback on the student’s own responses. We combined OPD with binary-reward RL.
The objective gradually shifted from pure reverse-KL distillation (β=1) toward binary-reward RL (β→0).
Stage 2 — PivotRL
PivotRL (Yi et al., 2026) is a turn-level RL method. It starts from existing SFT trajectories and treats each assistant turn as a possible training state. For each candidate turn, it samples several next actions from the frozen reference policy and scores them with a verifier. It keeps turns with mixed outcomes (some pass, some fail) and average reward below a difficulty threshold, discarding uniformly solved or uniformly failed turns. RL then samples fresh actions from the current policy at those retained “pivot” turns, using verifier rewards for locally valid actions rather than exact matches to the demonstration. On SWE-Bench, the paper reports accuracy comparable to end-to-end RL with about 4× fewer rollout turns.
We profiled 193 pivots from only 50 solved teacher trajectories. A functional-equivalence verifier checked whether each sampled tool call did the same job as the teacher’s example. For lookups, it compared behavior in the replayed environment; for actions that changed data, it compared tool arguments after standardizing their format. This let PivotRL reward equivalent actions even when their tool-call text differed from the teacher’s.
| PivotRL setting | Value |
|---|---|
| Loss | CISPO with asymmetric clipping [0.0, 4.0] |
| Advantages | Mean-centered: adv = r − mean(r), bounded |adv| ≤ 1; no standard-deviation normalization |
| Learning rate | 1e-5 |
| Pivot sampling | Beta-posterior sampling |
| Group size | 16 continuations per pivot (G=16) |
| Pivot batch size | 16 pivots per step (B=16) |
Reward over training
Evaluation Results
We evaluated each checkpoint on the same 97 τ³-Banking tasks, with five attempts per task and a 64k-token context window. GPT-5.4-mini simulated the user.
pass@5
pass@1 (avg 5)
General capability check
We evaluated the base model and our selected checkpoint on six reasoning, knowledge, instruction-following, and coding benchmarks using NeMo Evaluator, with thinking enabled and the same evaluation harness.
None of the six benchmarks showed a statistically significant improvement or decline.
Trained minus base score (percentage points)
GPQA-diamond
pass@1 (avg 3) · n=198
−0.34 pp95% confidence interval: −7.94 to +7.26 percentage points.MMLU-Pro
pass@1 (avg 2) · n=12,032
+0.04 pp95% confidence interval: −0.96 to +1.04 percentage points.AIME-2025
pass@1 (avg 8) · n=30
+1.25 pp95% confidence interval: −17.85 to +20.35 percentage points.IFEval
Prompt-strict · pass@1 (avg 2) · n=541
+0.18 pp95% confidence interval: −3.62 to +3.98 percentage points.SciCode
Subtask · pass@1 (avg 3) · n=338
+1.28 pp95% confidence interval: −5.62 to +8.18 percentage points.LiveCodeBench-v6
pass@1 (avg 3) · n=454
−1.84 pp95% confidence interval: −7.94 to +4.26 percentage points.
Get in touch here to access our off-the-shelf customer service tool use and agentic post-training datasets or reach out to us directly at research@afterquery.com.
AfterQuery is an applied research lab curating data solutions to accelerate foundation model development.
