AfterQuery closes $30M Series A at $300M valuation and surpasses $100M revenue run rate

Read blog
All benchmarks
Market-Bench

Introductory Quantitative Trading

Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics

Mean MAE · lower is better

Grok 4 (xAI)443.24
GPT-5.2 (OpenAI)969.39
Gemini 3 Pro Preview (Google)1744.27
GPT-5.1 Codex Max (OpenAI)4242.93
DeepSeek-V3.24575.64
Claude Sonnet 4.5 (Anthropic)5126.76
Claude Opus 4.5 (Anthropic)6039.62
Command A (Cohere)6562.11
Nova Premier (Amazon)7740.26
Llama 3.1 Nemotron Ultra (NVIDIA)9674.21
Llama 4 Maverick (Meta)10202.18
Mistral Large30605.53
Qwen3 Max (Alibaba)159144490.12

Overall Methodology

We split the strategies up into 3 different strategies: easy, medium, and hard. Each strategy has an associated prompt. In this prompt, we detail out the strategy specifics and task the model to build a backtester which correctly integrates the strategy and outputs the required metrics in CSV format.

This CSV is then compared to our verifiable backtester's outputs to calculate MAE. We also test how reliable the model is using pass@1 and pass@3 metrics.

All three strategies require models to track and reserve the liquidity they remove from the book through simulated trades. We create a synthetic book that nets the raw order book data with consumed liquidity. Strategies 2 and 3 also include a delay between submitting an order and hearing back from the exchange, mirroring the real world.

Simple Trading

Strategy 1 is a simple trading strategy where we task the model with buying and selling MSFT stock at predetermined times. We take Databento's market-by-price L10 data for MSFT. We then randomize the volume available at each price level to enforce correct synthetic-book reservation. If we schedule a trade for 10 shares and only 8 exist in the book, the model must only take 8. This caps the max volume available in the book.

The strategy tracks:

  • Cash and MSFT position
  • Realized P&L using FIFO accounting
  • Unrealized P&L based on raw-book mid-prices
  • An equity curve and maximum drawdown
  • Synthetic-book statistics such as total size available and bid/ask VWAP after model trades

Single-Stock Scheduled Execution

ModelMean MAEBest Run MAEExecutable Passes
gemini-3-pro-preview14.8314.835
claude-sonnet-4.516.3616.355
claude-opus-4.5111.407.183
grok-4121.857.224
deepseek-v3.2139.197.265
command-a184.90153.101
nova-premier-v1242.34153.102
mistral-large-2512361.8723.015
gpt-5.1-codex-max844.740.0025
gpt-5.21,545.89153.105
qwen3-max2,015.6616.395
llama-3.1-nemotron-ultra2,511.13111.633
llama-4-maverick4,137.62170.065

Pairs Trading

Strategy 2 is a pairs trading mean-reversion on Coke and Pepsi. At each book update, the model calculates a mid price for both symbols and forms a spread as a linear combination. A rolling history of the spread drives the mean and z-score signals across 3 states: flat, long-spread, or short-spread. When the z-score exceeds the entry threshold, it buys one leg and sells the other; positions flatten inside the exit threshold.

Additional features:

  • A cooldown to avoid rapid re-entry
  • A shared capital account
  • Synthetic books and VWAP tracking per symbol
  • Immediate-or-cancel limit orders priced from synthetic mid and spread

Pairs Mean-Reversion on COKE/PEP

ModelMean MAEBest Run MAEExecutable Passes
gemini-3-pro-preview52.2252.225
gpt-5.2107.0248.105
deepseek-v3.2132.77125.873
nova-premier-v1133.10133.100
claude-opus-4.5133.8685.562
gpt-5.1-codex-max136.9789.435
claude-sonnet-4.5193.3270.825
mistral-large-2512228.90133.105
grok-4309.37119.254
llama-4-maverick605.26131.922
command-a1,093.36133.102
llama-3.1-nemotron-ultra2,268.11133.102
qwen3-max408,991,460.86100.145

Dynamic Hedging

Strategy 3 is a realistic delta-hedging strategy using random walk deltas plus MSFT order book data. At regular intervals, it evaluates net delta and trades a portion to get flat, enforcing a minimum time between hedges. Hedge trades use fill-or-kill limit orders priced from synthetic mid and spread, with a fixed exchange delay before execution.

A synthetic book persists consumed liquidity. We track:

  • Stock position and options delta
  • Net delta of the combined portfolio
  • Realized and unrealized P&L from stock trades
  • Equity and maximum drawdown

Delta Hedging with MSFT

ModelMean MAEBest Run MAEExecutable Passes
gpt-5.21,369.591,365.425
grok-41,482.331,013.335
gemini-3-pro-preview4,595.521,013.995
gpt-5.1-codex-max10,496.401,370.725
deepseek-v3.214,724.361,127.145
claude-sonnet-4.516,157.31805.165
claude-opus-4.518,945.341,370.722
llama-4-maverick21,279.5220,418.581
nova-premier-v121,345.7521,345.750
llama-3.1-nemotron-ultra21,345.7521,345.750
command-a21,369.5819,852.902
mistral-large-251263,683.241,351.554
qwen3-max329,700.431,087.645