StoreBench: Evaluating AI Agents for Business Operations over Long Horizons

Business operations require a sequence of decisions under constraints on time, budget, and information. StoreBench evaluates whether AI agents can sustain these operations over long horizons by placing them in charge of Ostrelle, a simulated online apparel store built on Medusa v2. Agents purchase inventory, set prices, fulfill orders, and process returns as customer demand and supplier conditions change.

Agents use 29 merchant tools to manage these operations. They source inventory from a market of 40 suppliers and manage a catalog of 3,448 product variants, deciding what to stock, where to buy it, and how to allocate limited cash and actions.

The main evaluation covers 11 evaluation tasks, each lasting 30–45 simulated days. A separate experiment extends operation to a full year. Decisions have consequences across days: inventory purchases commit cash before sales arrive, late customer shipments can incur fines, and supplier failures can leave products unavailable.

StoreBench models an apparel store with simulated demand and scripted competitors. The framework could be extended to other business and operational settings.

The store runs on Medusa v2, a commerce platform. The agent has a limited number of operations in each time window. Reading a report uses an operation, just as shipping an order does; the store_status probe is free. Simulated time advances when the agent exhausts its operation budget or ends the window; model response speed does not affect it.

Vending-Bench was an important step toward evaluating agents that run a business over long horizons, testing whether they can manage a vending machine and grow its value. Vending-Bench 2 extends the task to a simulated year. However, a vending machine differs from an operating store in three ways that matter for evaluation and training: there is little to operate, the score has no reference scale, and language models simulate parts of the graded world. StoreBench addresses all three.

Unlike Vending-Bench, which tests web search, supplier negotiation, and (in Vending-Bench Arena) competition between agents, StoreBench uses scripted competitors and non-negotiating suppliers to focus on reproducible operations and calibrated evaluation.

The main suite evaluates thirteen models on 11 tasks lasting 30–45 simulated days. The leaderboard reports mean composite scores, pass rates, and component scores alongside scripted baselines and a human reference. A separate full-year experiment evaluates seven models, with four scoring between 0.966 and 0.976. Each tab shows the models evaluated under that protocol.

The main suite uses the goose harness without context compaction. The full-year experiment uses Claude Code with context compaction and extended thinking, different time windows, and no scripted shocks. Its scores are not directly comparable with main-suite scores.

These six scripted baselines use the same tools and operation budgets as the agents, providing reference points for scores and pass thresholds.

• smart-triage: uses hand-written decision rules to prioritize urgent store tasks.
• rule-based: runs the store using a fixed checklist of actions.
• blast-list: ships every order it can, without broader business decisions.
• absentee: attends each time window without taking store actions, earning only attendance credit (0.100).
• do-nothing: remains inactive, earning no attendance credit and scoring 0.000.
• exploiter: attempts known reward exploits to test whether the scoring safeguards hold.

Claude Fable 5.2 has the highest mean model score at 0.835 and the highest model pass rate at 89%. It is the only model clearly above smart-triage, which scores 0.764 and passes 97% of task–seed combinations: the gap is 0.071 (95% CI 0.045 to 0.099). GPT-6 Astra (0.772) and Claude Opus 5.5 (0.762) are within the margin of error of smart-triage. Model pass rates cover 99 runs each; the scripted baseline covers 33 combinations. All thirteen models score 1.000 for continuity, which measures attendance rather than successful store management.

These results show the average composite score for each evaluation task across all thirteen models. Each model value averages nine runs, with three attempts on each of three seeds. The human reference uses one selected outcome per evaluation task and seed, including two coached retries, with assignments favoring stronger participants. Its collection protocol differs from the three-attempt model evaluation; see the human-study details.

The composite score combines three component scores calculated from the store’s records. Each component and the composite range from 0 to 1; higher is better.

Composite score. Overall performance. The composite assigns a weight of 70% to business, 20% to reliability, and 10% to continuity. These weights define how much each component contributes to the score; they are not pass rates.

Pass rate. Share of runs that pass. The percentage of runs whose composite score exceeds the pass threshold. For each evaluation task and seed, the threshold is calculated from the scores of scripted comparison programs.

See the paper for scoring formulas and the full evaluation protocol.

Each evaluation task and seed has its own pass threshold. We score three scripted trading policies (smart-triage, rule-based, and blast-list) and set the threshold halfway between the two best scores. When the top two score within 0.02 of each other, the threshold sits 0.01 below the lower one. Thresholds range from 0.43 to 0.76, with a median of 0.72. The same threshold applies to every model and to the human reference.

smart-triage is the best of these policies on 31 of 33 combinations, so it clears the threshold almost by design: it passes 32 of 33. A pass rate therefore measures consistency as well as skill. smart-triage is deterministic and scores between 0.699 and 0.775 on every combination. Model runs vary more: the middle 80% of GPT-6 Astra’s runs span 0.662 to 0.888. A model can match or beat smart-triage on mean score and still fall below the threshold on some runs.

1. Compare against fixed rules. Programs that follow fixed rules set the score scale and pass threshold. Some do nothing; others follow a checklist or deal with urgent work first. Each evaluation task and seed gets its own scale.
2. Checks for reward hacking. Before scoring, the system settles bills, fines, and other obligations due during the run. Tests check known loopholes that could give an agent a high score without running the store well.
3. Replay the same decisions. Replaying the same recorded actions reproduces the financial result inside the environment.

These evaluation tasks test responses to higher demand, supplier disruption, and falling sales. Some events are announced or hinted at through rumors. Others must be inferred from changes in sales.

The failure analysis in the paper examines 693 main-suite runs across seven models: DeepSeek-V4-Pro, Claude Opus 4.8, Claude Fable 5, Qwen3.8-Max, Gemini 3.8 Flash, GPT-5.6 Sol, and Muse Spark 1.2. Across 11 tasks lasting 30–45 simulated days, three recurring failure modes involve how agents allocate their limited operating budget.

• Underbuying appears in 178 of 693 runs (26%), including runs from the strongest models in this analysis.
• Agents restock too late, buy too little, or overlook products outside the best sellers.
• In 60 runs they never place a purchase order; 41 of those are Gemini 3.8 Flash runs.
• Even a strong Claude Opus 4.8 run acknowledged that its less popular products were selling out without being restocked.

• Some agents offer less than the full refund.
• If the customer rejects it and the deadline passes, the store pays an ignored-return fine.
• Partial refunds can therefore increase total costs when unresolved returns incur fines.

• Muse Spark 1.2 and Gemini 3.8 Flash sometimes use shell scripts to send batches of shipping, refund, or restocking calls based on stale store data.

The separate full-year evaluation covers the same seven models and documents failures involving unsupported delegation and premature termination.

Unsupported delegation. In four of nine full-year attempts, Gemini 3.8 Flash claims that a background simulation process is operating the store and waits through the harness’s scheduling tool. These attempts have a mean composite score of approximately 0.1, compared with 0.799 in the five attempts where the agent operates the store.

Premature task completion. In one of nine attempts, GPT-5.6 Sol declares the task complete and stops at window 185 of 365. The remaining scheduled windows still count against its result.

If you’re a lab who wants us to evaluate your next model on StoreBench or access our off-the-shelf enterprise RL environments, get in touch here or reach out to us directly at research@afterquery.com.

AfterQuery is an applied research lab curating data solutions to accelerate foundation model development.