Introductory Quantitative Trading
Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics
Mean MAE · lower is better
Overall Methodology
We split the strategies up into 3 different strategies: easy, medium, and hard. Each strategy has an associated prompt. In this prompt, we detail out the strategy specifics and task the model to build a backtester which correctly integrates the strategy and outputs the required metrics in CSV format.
This CSV is then compared to our verifiable backtester's outputs to calculate MAE. We also test how reliable the model is using pass@1 and pass@3 metrics.
All three strategies require models to track and reserve the liquidity they remove from the book through simulated trades. We create a synthetic book that nets the raw order book data with consumed liquidity. Strategies 2 and 3 also include a delay between submitting an order and hearing back from the exchange, mirroring the real world.
Simple Trading
Strategy 1 is a simple trading strategy where we task the model with buying and selling MSFT stock at predetermined times. We take Databento's market-by-price L10 data for MSFT. We then randomize the volume available at each price level to enforce correct synthetic-book reservation. If we schedule a trade for 10 shares and only 8 exist in the book, the model must only take 8. This caps the max volume available in the book.
The strategy tracks:
- Cash and MSFT position
- Realized P&L using FIFO accounting
- Unrealized P&L based on raw-book mid-prices
- An equity curve and maximum drawdown
- Synthetic-book statistics such as total size available and bid/ask VWAP after model trades
Single-Stock Scheduled Execution
| Model | Mean MAE | Best Run MAE | Executable Passes |
|---|---|---|---|
| gemini-3-pro-preview | 14.83 | 14.83 | 5 |
| claude-sonnet-4.5 | 16.36 | 16.35 | 5 |
| claude-opus-4.5 | 111.40 | 7.18 | 3 |
| grok-4 | 121.85 | 7.22 | 4 |
| deepseek-v3.2 | 139.19 | 7.26 | 5 |
| command-a | 184.90 | 153.10 | 1 |
| nova-premier-v1 | 242.34 | 153.10 | 2 |
| mistral-large-2512 | 361.87 | 23.01 | 5 |
| gpt-5.1-codex-max | 844.74 | 0.002 | 5 |
| gpt-5.2 | 1,545.89 | 153.10 | 5 |
| qwen3-max | 2,015.66 | 16.39 | 5 |
| llama-3.1-nemotron-ultra | 2,511.13 | 111.63 | 3 |
| llama-4-maverick | 4,137.62 | 170.06 | 5 |
Pairs Trading
Strategy 2 is a pairs trading mean-reversion on Coke and Pepsi. At each book update, the model calculates a mid price for both symbols and forms a spread as a linear combination. A rolling history of the spread drives the mean and z-score signals across 3 states: flat, long-spread, or short-spread. When the z-score exceeds the entry threshold, it buys one leg and sells the other; positions flatten inside the exit threshold.
Additional features:
- A cooldown to avoid rapid re-entry
- A shared capital account
- Synthetic books and VWAP tracking per symbol
- Immediate-or-cancel limit orders priced from synthetic mid and spread
Pairs Mean-Reversion on COKE/PEP
| Model | Mean MAE | Best Run MAE | Executable Passes |
|---|---|---|---|
| gemini-3-pro-preview | 52.22 | 52.22 | 5 |
| gpt-5.2 | 107.02 | 48.10 | 5 |
| deepseek-v3.2 | 132.77 | 125.87 | 3 |
| nova-premier-v1 | 133.10 | 133.10 | 0 |
| claude-opus-4.5 | 133.86 | 85.56 | 2 |
| gpt-5.1-codex-max | 136.97 | 89.43 | 5 |
| claude-sonnet-4.5 | 193.32 | 70.82 | 5 |
| mistral-large-2512 | 228.90 | 133.10 | 5 |
| grok-4 | 309.37 | 119.25 | 4 |
| llama-4-maverick | 605.26 | 131.92 | 2 |
| command-a | 1,093.36 | 133.10 | 2 |
| llama-3.1-nemotron-ultra | 2,268.11 | 133.10 | 2 |
| qwen3-max | 408,991,460.86 | 100.14 | 5 |
Dynamic Hedging
Strategy 3 is a realistic delta-hedging strategy using random walk deltas plus MSFT order book data. At regular intervals, it evaluates net delta and trades a portion to get flat, enforcing a minimum time between hedges. Hedge trades use fill-or-kill limit orders priced from synthetic mid and spread, with a fixed exchange delay before execution.
A synthetic book persists consumed liquidity. We track:
- Stock position and options delta
- Net delta of the combined portfolio
- Realized and unrealized P&L from stock trades
- Equity and maximum drawdown
Delta Hedging with MSFT
| Model | Mean MAE | Best Run MAE | Executable Passes |
|---|---|---|---|
| gpt-5.2 | 1,369.59 | 1,365.42 | 5 |
| grok-4 | 1,482.33 | 1,013.33 | 5 |
| gemini-3-pro-preview | 4,595.52 | 1,013.99 | 5 |
| gpt-5.1-codex-max | 10,496.40 | 1,370.72 | 5 |
| deepseek-v3.2 | 14,724.36 | 1,127.14 | 5 |
| claude-sonnet-4.5 | 16,157.31 | 805.16 | 5 |
| claude-opus-4.5 | 18,945.34 | 1,370.72 | 2 |
| llama-4-maverick | 21,279.52 | 20,418.58 | 1 |
| nova-premier-v1 | 21,345.75 | 21,345.75 | 0 |
| llama-3.1-nemotron-ultra | 21,345.75 | 21,345.75 | 0 |
| command-a | 21,369.58 | 19,852.90 | 2 |
| mistral-large-2512 | 63,683.24 | 1,351.55 | 4 |
| qwen3-max | 329,700.43 | 1,087.64 | 5 |