Blog

Improving Nemotron and Qwen on GDP.pdf with xToken

Michael E.Ricky R.Nathan J.

Introduction

We trained Qwen3.5-9B and Nemotron 3.5 Super VL on 538 professional document tasks using NVIDIA’s xToken, a cross-tokenizer distillation method introduced by Sreenivas et al. (2026). On public GDP.pdf tasks held out from training, Qwen3.5-9B’s strict pass@1 increased from 4.40% to a peak of 8.20%. Nemotron 3.5 Super VL’s rose from 6.67% to 11.83%.

GDP.pdf is Surge AI’s benchmark for grounded multimodal reasoning over professional PDFs. Each task uses a single PDF and asks for an answer, such as a short explanation, list, or table, generated in one model call.

The original xToken study focused on continued pre-training of a 1B base model. Later work by Liu et al. (2026) evaluated it as a baseline on math and code tasks with the post-trained Qwen3.5-2B. We build on our earlier professional work experiments to apply xToken to instruction-tuned students on professional document tasks.

We used a modified version of the off-policy xToken recipe in NeMo RL to support long document inputs with images, cached teacher predictions, and teacher selection based on rubric performance.

Training data

We trained on 538 tasks from AfterQuery’s professional document dataset. AfterQuery domain experts from the relevant industries authored the tasks and their rubrics: healthcare, engineering, legal, STEM and research, insurance, human resources, construction, real estate, manufacturing and supply chains, and finance and investing.

Each task pairs a professional question with a source PDF and a rubric specifying what a successful answer must contain. The selected tasks cover 535 distinct PDFs and 7,435 rubric criteria, with 5–30 criteria per task.

The work ranges from reconciling Medicare documentation requirements after a rule change to checking whether four concrete mixes meet a strength requirement, reading reactor-material properties from a chart, and explaining missing values in an industrial supplier report.

Why xToken

In our previous professional work experiments, we used on-policy distillation (OPD) to train smaller student models with feedback from stronger teachers. In OPD, the student generates a response to a task, and the teacher assigns a probability to each token given the task and the preceding text. Training uses these probabilities to guide the student toward choices the teacher considers more likely.

Our student and teacher shared a tokenizer, which splits text into pieces called tokens. Because both models used the same pieces, we could compare their next-token probabilities directly.

That direct comparison breaks down when the models split text differently. For example, Nemotron 3.5 Super VL tokenizes Footnote as one token, while Qwen3.8-27B splits it into Foot and note. One student token now corresponds to two teacher tokens, so their predictions no longer line up.

To make this comparison possible, we used xToken to align groups of tokens that cover the same text and connect the two vocabularies through a weighted map, W. In the Footnote example, W divides the student’s probability for Footnote between the teacher’s Foot and note entries, with the weights determining how much goes to each. Applying this map across the vocabulary lets us compare the two models’ probabilities for the same token entries.

This comparison provides the training signal: projected forward KL (P-KL) penalizes the student when its mapped prediction gives too little probability to choices the teacher considers likely. With the teacher and map held fixed, training uses that signal to update the student’s weights, bringing its mapped predictions closer to the teacher’s while leaving its tokenizer unchanged.

The Nemotron 3.5 Super VL student tokenizer splits Footnote as [Footnote], while the Qwen3.8-27B teacher tokenizer splits it as [Foot] [note]. The tokens move into vocabulary map W: student tokens label rows and teacher tokens label columns. Report and summary map to exact matches with weight 1; Footnote maps to Foot with weight 0.9091 and note with weight 0.0909.

Why not just use SFT with the teacher traces?

Supervised fine-tuning (SFT) teaches the student to reproduce what the teacher wrote, using the student’s own tokenizer. We also want the student to learn how the teacher weighed the alternatives, which is what xToken adds.

For example, suppose the teacher’s next-token probabilities are 60% for report, 30% for summary, and 10% for everything else. If it chooses report, SFT uses that token as the target.

With xToken, the student can also learn from the 30% probability assigned to summary, even though the teacher chose report.

Our experiment

Both students learned from three teachers: Qwen3.8-27B, DeepSeek-V4.1-Flash, and Kimi-K3. The Qwen student and teacher share the same text tokenizer; DeepSeek and Kimi provide feedback across tokenizers.

We generated candidate teacher responses and checked them against each task’s rubric. A teacher qualified for a task only when its answer passed every criterion. We selected a complete response as the training target and gave the qualifying teachers equal weight in that task’s distillation loss.

While the original xToken paper selects standard KL, P-KL, or H-KL for each teacher–student pair, we used P-KL for all three teachers and did not compare the alternative loss formulations.

Within each aligned span, we multiplied the token probabilities and normalized the result before applying the teacher’s vocabulary map. We computed a separate P-KL loss for each qualifying teacher on the same training response and averaged these losses per task. Across the training batch, we balanced this distillation loss with the loss for reproducing the response, then updated only the student.

Training curves

Qwen3.5-9B · Next-token training accuracy

75%80%85%90%95%1204066Training step79.08%179.20%278.04%379.31%479.41%579.69%679.04%780.92%882.60%980.58%1082.55%1180.77%1280.21%1381.19%1481.58%1581.35%1679.38%1782.58%1883.26%1983.97%2082.73%2184.99%2283.64%2383.47%2483.05%2582.77%2684.71%2783.45%2883.89%2984.42%3081.89%3183.73%3282.62%3384.55%3484.59%3586.57%3685.59%3786.01%3885.88%3986.08%4086.13%4186.63%4287.01%4386.87%4486.14%4587.50%4687.41%4787.67%4886.58%4987.58%5085.61%5189.73%5288.91%5388.54%5489.38%5588.68%5689.66%5789.88%5889.92%5990.92%6089.80%6189.81%6287.96%6390.62%6590.00%66

Nemotron 3.5 Super VL · Next-token training accuracy

80%81%82%83%84%85%86%87%25101521Training step82.65%281.74%382.35%482.06%582.31%681.88%783.36%884.57%982.67%1084.49%1182.82%1282.54%1383.12%1483.64%1583.51%1681.47%1785.10%1885.59%1986.10%2084.89%21
Next-token accuracy measures how often the student’s top prediction matches the next token in the training response. Qwen3.5-9B rose from 79.08% to 90.00%; Nemotron 3.5 Super VL rose from 82.65% at step 2 to 84.89% at step 21.

Evaluation methodology

How we evaluate with NeMo Gym

We use the GDP.pdf implementation in NVIDIA’s NeMo Gym repository, pinned to commit 8908cb2. We evaluate each base model and its trained checkpoints with the same document agent, rubric verifier, and judge prompt. Each comparison uses the same tasks, with all five attempts graded at every compared checkpoint.

Document input and generation

We prepare each task’s PDF with LiteParse text extraction and OCR, plus page images at 150 DPI. Gym’s GDP.pdf agent sends the task prompt, extracted text, and images in a single turn without tools, adapting image resolution and page layout to the model’s input limits.

We sample five answers per task with thinking enabled, keeping these settings fixed within each model family.

Generation settings for the GDP.pdf evaluation
SettingQwen3.5-9BNemotron 3.5 Super VL
Attempts per task55
Temperature1.00.6
Top-p0.950.95
Maximum output tokens32,76820,480

Rubric grading and metrics

Gym’s rubric verifier grades each criterion with openai/gpt-5.6-luna through OpenRouter. The judge receives the task prompt, final answer, and one criterion, and returns a binary score with a rationale. It receives neither the PDF nor the model’s separate reasoning output.

  • Criteria score: the fraction of rubric criteria passed in each answer, averaged over the five attempts and then over tasks. Every task has equal weight.
  • Strict pass@1: the fraction of answers that pass every criterion, averaged over five attempts per task.
  • Strict pass@5: the fraction of tasks with at least one fully correct answer among the five attempts.

The gains we saw

We evaluated both students before and after training on held-out GDP.pdf tasks, which test reasoning over professional documents. We generated five answers per task and counted an answer as a strict pass only when it met every rubric criterion. Strict pass@1 avg5 is the average success rate across those attempts; strict pass@5 is the share of tasks with at least one passing answer.

BaseAfter xToken

Strict pass@1 avg 5. Qwen3.5-9B: Base 4.40% with 8,935 mean output tokens; step 64 8.20% with 15,997 mean output tokens. Gain 3.80 percentage points and 79.0% more output tokens. Nemotron 3.5 Super VL: Base 6.67% with 7,645 mean output tokens; step 21 11.83% with 13,285 mean output tokens. Gain 5.16 percentage points and 73.8% more output tokens.

Strict pass@1 avg 5 versus mean output tokensQwen3.5-9B: Base 4.40% with 8,935 mean output tokens; step 64 8.20% with 15,997 mean output tokens. Gain 3.80 percentage points and 79.0% more output tokens. Nemotron 3.5 Super VL: Base 6.67% with 7,645 mean output tokens; step 21 11.83% with 13,285 mean output tokens. Gain 5.16 percentage points and 73.8% more output tokens.Strict pass@1 avg 5 (%)Mean output tokens per answerQwen3.5-9B: Base → step 64Base · 4.40%8,935 tokens8.20%15,997 tokensNemotron 3.5 Super VL: Base → step 21Base · 6.67%7,645 tokens11.83%13,285 tokens
Qwen3.5-9B

+3.80 pts

+7,062 output tokens (+79.0%)

Nemotron 3.5 Super VL

+5.16 pts

+5,640 output tokens (+73.8%)

Both models improved strict pass@1 and pass@5 after P-KL training with a pool of three teachers. Their average outputs also grew by roughly 74–80%, so the gains came with higher token use.

What changed in the answers

Which dental plans cover same-name counties in Minnesota and Wisconsin? Base Nemotron conflated the two Douglas counties. After training, it assigned the correct plan to each.

Dental-plan map from the task document. Douglas County, Minnesota, is shaded for Delta Dental Premier; Douglas County, Wisconsin, is shaded for Delta Dental PPO. Polk County is shaded for Premier in both states.
Task document: University of Minnesota, New Employees Guide for Benefits Enrollment, p. 20.
Base modelRubric score 71.4%

Douglas County: Delta Dental PPO

Polk County: Delta Dental Premier

Pierce County: Delta Dental PPO

St. Croix County: Delta Dental PPO

Post-trained modelRubric score 100%

Douglas County, Minnesota — Base dental plan: Delta Dental Premier (Greater Minnesota base plan).

Douglas County, Wisconsin — Base dental plan: Delta Dental PPO (Duluth‑area / border‑community base plan).

Another task asked Nemotron to calculate an insurance premium. The base answer noted the earthquake exception, then discounted both coverages in a later step. The trained answer kept earthquake coverage separate through the final calculation.

Rating Modification section of the insurance manual, stating that Individual Risk Premium Modification does not apply to earthquake coverages.
Task document: Acadia Insurance Company, Commercial Lines Manual (Georgia), p. 8.
Base modelRubric score 75%

Total Premium before expense adjustment: $1,870 (Property) + $1,000 (Earthquake) = $2,870

Calculation: $2,870 × 0.909 = $2,608.83

Post-trained modelRubric score 100%

The expense reduction is part of the Rating Modification section, so it is applied to the property premium (the portion subject to the Individual Risk Premium Modification):

$1,870.00 × 0.909 = $1,699.83

Final total policy premium: $2,699.83 (Property $1,699.83 + Earthquake $1,000.00).

Get in touch here to access our off-the-shelf knowledge work agent training datasets or reach out to us directly at research@afterquery.com to see the full post-training experiment.

AfterQuery is an applied research lab curating data solutions to accelerate foundation model development.

Notes and sources

  1. Sharath Turuvekere Sreenivas, Adithyakrishna Venkatesh Hanasoge, Mingyu Yang, Ali Taghibakhshi, Saurav Muralidharan, Ashwath Aithal, and Pavlo Molchanov. (2026). X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation. NVIDIA. arXiv:2605.21699.
  2. Jiacheng Liu, Jingwei Song, Qituan Zhang, Siheng Chen, and Linfeng Zhang. (2026). Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation. arXiv:2609.34738.
  3. Qwen. Qwen3.5-2B model card. Hugging Face.
  4. Suhaas Garre, Emily Ritchie, Sushant Mehta, and Edwin Chen. (2026). GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents. arXiv:2607.11192.
  5. NeMo RL: Cross-Tokenizer Off-Policy Distillation.
  6. NVIDIA. NeMo Gym: GDP.pdf benchmark, document agent, and rubric verifier.
  7. AfterQuery’s earlier on-policy distillation experiments on GDPval.

Cite this post

BibTeX
@misc{afterquery2026xtoken,
  author = {{Michael E.} and {Ricky R.} and {Nathan J.}},
  title = {{Improving Nemotron and Qwen on GDP.pdf with xToken}},
  year = {2026},
  month = oct,
  howpublished = {AfterQuery},
  url = {https://www.afterquery.com/blog/xtoken-distillation-gdp-pdf}
}

Related articles