
Improving Nemotron and Qwen on GDP.pdf with xToken
Michael E.
Ricky R.
Nathan J.- Base
- Trained
- Base
- Trained
Introduction
We trained Qwen3.5-9B and Nemotron 3.5 Super VL on 538 professional document tasks using NVIDIA’s xToken, a cross-tokenizer distillation method introduced by Sreenivas et al. (2026). On public GDP.pdf tasks held out from training, Qwen3.5-9B’s strict pass@1 increased from 4.40% to a peak of 8.20%. Nemotron 3.5 Super VL’s rose from 6.67% to 11.83%.
GDP.pdf is Surge AI’s benchmark for grounded multimodal reasoning over professional PDFs. Each task uses a single PDF and asks for an answer, such as a short explanation, list, or table, generated in one model call.
The original xToken study focused on continued pre-training of a 1B base model. Later work by Liu et al. (2026) evaluated it as a baseline on math and code tasks with the post-trained Qwen3.5-2B. We build on our earlier professional work experiments to apply xToken to instruction-tuned students on professional document tasks.
We used a modified version of the off-policy xToken recipe in NeMo RL to support long document inputs with images, cached teacher predictions, and teacher selection based on rubric performance.
Training data
We trained on 538 tasks from AfterQuery’s professional document dataset. AfterQuery domain experts from the relevant industries authored the tasks and their rubrics: healthcare, engineering, legal, STEM and research, insurance, human resources, construction, real estate, manufacturing and supply chains, and finance and investing.
Each task pairs a professional question with a source PDF and a rubric specifying what a successful answer must contain. The selected tasks cover 535 distinct PDFs and 7,435 rubric criteria, with 5–30 criteria per task.
The work ranges from reconciling Medicare documentation requirements after a rule change to checking whether four concrete mixes meet a strength requirement, reading reactor-material properties from a chart, and explaining missing values in an industrial supplier report.
- Healthcare10619.7%
- Engineering6812.6%
- Legal6411.9%
- STEM and research6111.3%
- Insurance5810.8%
- Human resources529.7%
- Construction478.7%
- Real estate346.3%
- Mfg. / supply chains285.2%
- Finance and investing203.7%
Why xToken
In our previous professional work experiments, we used on-policy distillation (OPD) to train smaller student models with feedback from stronger teachers. In OPD, the student generates a response to a task, and the teacher assigns a probability to each token given the task and the preceding text. Training uses these probabilities to guide the student toward choices the teacher considers more likely.
Our student and teacher shared a tokenizer, which splits text into pieces called tokens. Because both models used the same pieces, we could compare their next-token probabilities directly.
That direct comparison breaks down when the models split text differently. For example, Nemotron 3.5 Super VL tokenizes Footnote as one token, while Qwen3.8-27B splits it into Foot and note. One student token now corresponds to two teacher tokens, so their predictions no longer line up.
To make this comparison possible, we used xToken to align groups of tokens that cover the same text and connect the two vocabularies through a weighted map, W. In the Footnote example, W divides the student’s probability for Footnote between the teacher’s Foot and note entries, with the weights determining how much goes to each. Applying this map across the vocabulary lets us compare the two models’ probabilities for the same token entries.
This comparison provides the training signal: projected forward KL (P-KL) penalizes the student when its mapped prediction gives too little probability to choices the teacher considers likely. With the teacher and map held fixed, training uses that signal to update the student’s weights, bringing its mapped predictions closer to the teacher’s while leaving its tokenizer unchanged.
![The Nemotron 3.5 Super VL student tokenizer splits Footnote as [Footnote], while the Qwen3.8-27B teacher tokenizer splits it as [Foot] [note]. The tokens move into vocabulary map W: student tokens label rows and teacher tokens label columns. Report and summary map to exact matches with weight 1; Footnote maps to Foot with weight 0.9091 and note with weight 0.0909.](/images/blog/xtoken/tokenizer-to-matrix.gif?dpl=dpl_CfVEmdn51kuiVfpYUE5qvM2FvKks)
Why not just use SFT with the teacher traces?
Supervised fine-tuning (SFT) teaches the student to reproduce what the teacher wrote, using the student’s own tokenizer. We also want the student to learn how the teacher weighed the alternatives, which is what xToken adds.
For example, suppose the teacher’s next-token probabilities are 60% for report, 30% for summary, and 10% for everything else. If it chooses report, SFT uses that token as the target.
With xToken, the student can also learn from the 30% probability assigned to summary, even though the teacher chose report.
Our experiment
Both students learned from three teachers: Qwen3.8-27B, DeepSeek-V4.1-Flash, and Kimi-K3. The Qwen student and teacher share the same text tokenizer; DeepSeek and Kimi provide feedback across tokenizers.
We generated candidate teacher responses and checked them against each task’s rubric. A teacher qualified for a task only when its answer passed every criterion. We selected a complete response as the training target and gave the qualifying teachers equal weight in that task’s distillation loss.
While the original xToken paper selects standard KL, P-KL, or H-KL for each teacher–student pair, we used P-KL for all three teachers and did not compare the alternative loss formulations.
Within each aligned span, we multiplied the token probabilities and normalized the result before applying the teacher’s vocabulary map. We computed a separate P-KL loss for each qualifying teacher on the same training response and averaged these losses per task. Across the training batch, we balanced this distillation loss with the loss for reproducing the response, then updated only the student.
Training curves
Qwen3.5-9B · Next-token training accuracy
Nemotron 3.5 Super VL · Next-token training accuracy
Evaluation methodology
How we evaluate with NeMo Gym
We use the GDP.pdf implementation in NVIDIA’s NeMo Gym repository, pinned to commit 8908cb2. We evaluate each base model and its trained checkpoints with the same document agent, rubric verifier, and judge prompt. Each comparison uses the same tasks, with all five attempts graded at every compared checkpoint.
Document input and generation
We prepare each task’s PDF with LiteParse text extraction and OCR, plus page images at 150 DPI. Gym’s GDP.pdf agent sends the task prompt, extracted text, and images in a single turn without tools, adapting image resolution and page layout to the model’s input limits.
We sample five answers per task with thinking enabled, keeping these settings fixed within each model family.
| Setting | Qwen3.5-9B | Nemotron 3.5 Super VL |
|---|---|---|
| Attempts per task | 5 | 5 |
| Temperature | 1.0 | 0.6 |
| Top-p | 0.95 | 0.95 |
| Maximum output tokens | 32,768 | 20,480 |
Rubric grading and metrics
Gym’s rubric verifier grades each criterion with openai/gpt-5.6-luna through OpenRouter. The judge receives the task prompt, final answer, and one criterion, and returns a binary score with a rationale. It receives neither the PDF nor the model’s separate reasoning output.
- Criteria score: the fraction of rubric criteria passed in each answer, averaged over the five attempts and then over tasks. Every task has equal weight.
- Strict pass@1: the fraction of answers that pass every criterion, averaged over five attempts per task.
- Strict pass@5: the fraction of tasks with at least one fully correct answer among the five attempts.
The gains we saw
We evaluated both students before and after training on held-out GDP.pdf tasks, which test reasoning over professional documents. We generated five answers per task and counted an answer as a strict pass only when it met every rubric criterion. Strict pass@1 avg5 is the average success rate across those attempts; strict pass@5 is the share of tasks with at least one passing answer.
Strict pass@1 avg 5. Qwen3.5-9B: Base 4.40% with 8,935 mean output tokens; step 64 8.20% with 15,997 mean output tokens. Gain 3.80 percentage points and 79.0% more output tokens. Nemotron 3.5 Super VL: Base 6.67% with 7,645 mean output tokens; step 21 11.83% with 13,285 mean output tokens. Gain 5.16 percentage points and 73.8% more output tokens.
+3.80 pts
+7,062 output tokens (+79.0%)
+5.16 pts
+5,640 output tokens (+73.8%)
Both models improved strict pass@1 and pass@5 after P-KL training with a pool of three teachers. Their average outputs also grew by roughly 74–80%, so the gains came with higher token use.
What changed in the answers
Which dental plans cover same-name counties in Minnesota and Wisconsin? Base Nemotron conflated the two Douglas counties. After training, it assigned the correct plan to each.

Douglas County: Delta Dental PPO
Polk County: Delta Dental Premier
Pierce County: Delta Dental PPO
St. Croix County: Delta Dental PPO
Douglas County, Minnesota — Base dental plan: Delta Dental Premier (Greater Minnesota base plan).
Douglas County, Wisconsin — Base dental plan: Delta Dental PPO (Duluth‑area / border‑community base plan).
Another task asked Nemotron to calculate an insurance premium. The base answer noted the earthquake exception, then discounted both coverages in a later step. The trained answer kept earthquake coverage separate through the final calculation.

Total Premium before expense adjustment: $1,870 (Property) + $1,000 (Earthquake) = $2,870
Calculation: $2,870 × 0.909 = $2,608.83
The expense reduction is part of the Rating Modification section, so it is applied to the property premium (the portion subject to the Individual Risk Premium Modification):
$1,870.00 × 0.909 = $1,699.83
Final total policy premium: $2,699.83 (Property $1,699.83 + Earthquake $1,000.00).
Get in touch here to access our off-the-shelf knowledge work agent training datasets or reach out to us directly at research@afterquery.com to see the full post-training experiment.
AfterQuery is an applied research lab curating data solutions to accelerate foundation model development.
Notes and sources
- Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, and Pavlo Molchanov. (2026). X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation. NVIDIA. arXiv:2605.21699.
- Jiacheng Liu, Jingwei Song, Qituan Zhang, Siheng Chen, and Linfeng Zhang. (2026). Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation. arXiv:2609.34738.
- Qwen. Qwen3.5-2B model card. Hugging Face.
- Suhaas Garre, Emily Ritchie, Sushant Mehta, and Edwin Chen. (2026). GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents. arXiv:2607.11192.
- NeMo RL: Cross-Tokenizer Off-Policy Distillation.
- NVIDIA. NeMo Gym: GDP.pdf benchmark, document agent, and rubric verifier.
- AfterQuery’s earlier on-policy distillation experiments on GDPval.
Cite this post
@misc{afterquery2026xtoken,
author = {{Michael E.} and {Ricky R.} and {Nathan J.}},
title = {{Improving Nemotron and Qwen on GDP.pdf with xToken}},
year = {2026},
month = oct,
howpublished = {AfterQuery},
url = {https://www.afterquery.com/blog/xtoken-distillation-gdp-pdf}
}


