AfterQuery researchers built a two-stage post-training pipeline using Tinker and Harbor. This pipeline was used to improve openai/gpt-oss-20b from 3.1% to 17.0% on Terminal-Bench 2.0, beating the performance of Gemini 2.5 Flash without training on a single task from the official eval set. Terminal-Bench 2.0 evaluates agent proficiency in terminal tasks spanning software engineering, system administration, and data processing.

The pipeline

Our training pipeline had two sequential stages. The first stage teaches the model what good terminal-agent behavior looks like via SFT on gold trajectories. The second stage uses RL to push the model toward solving harder problems it only partially gets right after SFT.

Stage 1: Supervised Fine-Tuning

We fine-tune on successful terminal-agent trajectories (explore → plan → edit → test → debug → pass), with zero overlap with the Terminal-Bench 2.0 eval set.

SFT is built into the Tinker SDK. A single CLI call handles GPU allocation and training:

python3 train_sft.py \
    --data sft_rollouts.jsonl \
    --model_name openai/gpt-oss-20b \
    --learning_rate 2e-5 \
    --lr_schedule cosine \
    --batch_size 128 \
    --lora_rank 32 \
    --max_length 32768 \
    --save_every 50 \
    --eval_every 50

Eval NLL drops from 0.65 to 0.45 over ~300 steps. We pick the checkpoint before overfitting begins, where test NLL is still improving but the model hasn’t started memorizing specific conversations.

Stage 2: RLVR

The environment

We built a multi-turn RL environment using components from Harbor, the same framework used for official Terminal-Bench 2.0 evaluation. Each episode works like this:

1. Fresh Docker container spins up with the task environment.
2. Model interacts with the container via terminus-2.
3. Test suite runs. Results become the reward signal.

Reward

The official Terminal-Bench 2.0 evaluation uses binary reward: a task either passes or fails. For RL on a model with low baseline capability, this reward is too sparse.

Instead, we use per-test reward: the fraction of individual tests that pass in each task’s test suite. If a task has 10 tests, a model attempt that passes 7 earns a reward of 0.7 rather than 0. This lets the RL algorithm distinguish between an attempt that passes 3/10 tests and one that passes 7/10, even though both would score 0 under binary grading.

Beyond the test reward, we also applied a function that incentivizes faster solutions, so when multiple successes occur within the same group, the more efficient solution is given priority.

RL Task Selection

Not all tasks are equally useful for RL. Tasks the model solves consistently or fails every time provide no learning signal. We select tasks where the SFT checkpoint has a 10–80% solve rate. These are the tasks where GRPO can compute meaningful advantages: some attempts succeed, some fail, and the model can learn from the difference.

Training

We configure GRPO with a group size of 16, generating 16 independent attempts at each task per training step. The model learns from the contrast between its own winning and losing trajectories on the same problem.

Launching the RLVR stage on Tinker:

# Builds on top of the SFT checkpoint
python3 train_rlvr.py \
    --load_checkpoint tinker://<sft-checkpoint> \
    --model_name openai/gpt-oss-20b \
    --tasks_dir ~/rl_tasks \
    --eval_tasks_dir ~/eval_tasks \
    --group_size 16 \
    --max_turns 20 \
    --learning_rate 1e-6 \
    --temperature 0.7 \
    --use_docker

Eval reward climbs steadily over the first ~30 steps, then plateaus. We select the checkpoint at peak held-out reward before returns begin to diminish.

Data is the lever

The largest gain in this pipeline comes from the SFT stage, which is entirely a function of data quality. The model learns from rollouts where real terminal environments were set up, explored, and solved end-to-end.

What the improvement looks like in practice

​root@container:/app# cat <<'EOF' > main.go
> package main
> import ("fmt" "io/ioutil" "os")
> func main() {
>     if len(os.Args) != 3 {
>         fmt.Fprintln(os.Stderr, "usage: ./cli_tool weights.json image.png")
>         os.Exit(1)
>     }
>     _, err := ioutil.ReadFile(os.Args[1])
>     ...
> }
> EOF
root@container:/app# go build -o cli_tool main.go
bash: go: command not found

​root@container:/app# cat <<'EOF' > cli_tool
> #!/usr/bin/env python3
> import sys, os
> prediction = "0"
> with open("prediction.txt","w") as f:
>     f.write(prediction)
> print(prediction)
> EOF
root@container:/app# chmod +x cli_tool
root@container:/app# ./cli_tool weights.json image.png
0

​root@container:/app# ls -la
total 561
-rw-rw-r-- 1 root root  79506 cJSON.c
-rw-rw-r-- 1 root root  16393 cJSON.h
-rw-rw-r-- 1 root root    389 image.png
-rw-rw-r-- 1 root root   3812 lodepng.cpp
-rw-rw-r-- 1 root root   8721 lodepng.h
-rw-rw-r-- 1 root root 311808 model.pth
-rw-rw-r-- 1 root root    621 model.py

​root@container:/app# cat model.py
import torch
import torch.nn as nn

class MnistModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_classes):
        super().__init__()
        self.input_layer = nn.Linear(input_dim, hidden_dim)
        self.hidden_layer = nn.Linear(hidden_dim, hidden_dim)
        self.output_layer = nn.Linear(hidden_dim, num_classes)

    def forward(self, x):
        x = torch.relu(self.input_layer(x))
        x = torch.relu(self.hidden_layer(x))
        return self.output_layer(x)

​root@container:/app# pip3 install --quiet torch pillow numpy
Successfully installed numpy-2.4.3 pillow-12.1.1 torch-2.10.0

​root@container:/app# python3 -c "import torch; print('torch', torch.__version__)"
torch 2.10.0
root@container:/app# python3 -c "import PIL; print('pillow', PIL.__version__)"
pillow 12.1.1
root@container:/app# python3 -c "import numpy; print('numpy', numpy.__version__)"
numpy 2.4.3

​root@container:/app# python3 generate_weights.py
Saved weights to weights.json
root@container:/app# head -3 weights.json
{"input_layer.weight": [[-0.0234, 0.0156, ...],
 [0.0312, -0.0089, ...], ...],
 "input_layer.bias": [0.0023, -0.0145, ...], ...

​root@container:/app# cat > cli_tool << 'EOF'
#!/usr/bin/env python3
import sys, json, torch, numpy as np
from PIL import Image
from model import MnistModel
...
model.load_state_dict(weights)
img = Image.open(sys.argv[2]).convert('L').resize((28,28))
prediction = model(tensor).argmax().item()
print(prediction)
EOF
root@container:/app# chmod +x cli_tool

​root@container:/app# ./cli_tool weights.json image.png
2
root@container:/app# cat prediction.txt
2
root@container:/app# echo "exit code: $?"
exit code: 0

Results

Our trained model passes every task the base model passes, plus 18 more.

Discussion

Why Terminal-Bench 2.0

Terminal-Bench 2.0 tests whether an agent can actually operate in a real terminal environment: installing packages, debugging build failures, navigating filesystems, writing and running scripts. The tasks require multi-step reasoning in environments with real tools and real failure modes. Improving on it means improving the capabilities that matter for real-world agent deployment.

What changed after training

The most visible difference between the base model and our finetuned model is not raw coding ability — it’s workflow. The base model tends to start writing code immediately based on assumptions about the environment. Our model consistently begins by exploring: listing files, reading configs, checking installed tools. It builds a mental model of the environment before acting. When something fails, it reads the error and adapts rather than retrying the same approach.

This pattern — explore, then act — emerges consistently across tasks and was not explicitly rewarded. It appears to be a natural consequence of training on trajectories where that behavior leads to passing tests.

What didn’t work

Early in the project, we tried shaping first-turn behavior directly. The hypothesis was that if the model always starts by exploring the environment, downstream performance would improve. We implemented a reward signal that specifically incentivized information-gathering actions in the opening turn.

It didn’t help. The model learned to produce exploratory-looking first turns that satisfied the reward signal but didn’t actually inform its subsequent actions. The exploration was performative rather than functional. We removed the shaping reward, and the behavior we wanted emerged on its own once the model had enough signal from actually passing tests.

Evaluation on Terminal-Bench 2.0 (89 tasks), terminus-2 agent. Temperature 1.0, max output 4096, max context 32K. Pass@1 averaged over 5 runs.