CrossSearch: A Benchmark for Agents That Cross-Reference Local Files and the Web
Sam J.Overview
Much of modern professional work involves checking local records against sources on the web. For example, a technician may need to match thermal readings to a camera’s spec sheet, a developer may match a project’s lockfile to a security advisory, or a city planner may match an intersection’s traffic counts to a federal manual for traffic signals. Agents doing this work often operate in real workspaces, where facts are spread across documents, spreadsheets and photos. They also need the open web, because the public source the answer depends on is usually not in those files. Agents need this dual capability: reading files to know what to search for, then applying what they find on the web back to the workspace.
We introduce CrossSearch, a benchmark that measures agents’ ability to cross-reference sources in local workspaces and on the web. We release 100 tasks created by domain professionals. These cover areas such as aircraft maintenance, pharmacy, tax, and software security. For each task, the agent receives three things: (1) a real-world request, written the way a colleague would ask for it, (2) a folder with the actual files as they were provided (in any format), and (3) access to the open web. Each of the tasks also has at least one public source that is not explicitly mentioned in the request. The agent has to work out what to search for from the files. We structured the tasks so that they can’t be answered from just the web or just the local workspace.
We believe the same capability is needed to generate high-fidelity synthetic workspaces, because as a workspace grows, its files and facts have to stay coherent for synthetic post-training to scale.
Results
We report pass^3, the share of tasks where all three attempts receive full credit, and pass@1, the share of attempts that receive full credit.
CrossSearch leaderboard
- Gemini 4 Argon64.0%
- GPT-6 Astra60.0%
- GPT-6.1 Sol59.0%
- Opus 5.557.0%
- GPT-5.6 Sol38.0%
- Grok 4.635.0%
- Opus 534.0%
- Gemini 3.8 Flash29.0%
- Inkling 2 Small23.0%
- Kimi K315.0%
- Sonnet 57.0%
- Inkling0.0%
An example task
Starting workspace
Is the ordered dose right for this patient? If not, what should it be, and what is the main risk?
Open web
Full credit = all seven rubric items. The score counts how many were met.
- C1age 79, 59 kg, creatinine 1.7
- C2indication is nonvalvular AF
- C3exactly 2 of 3 criteria met
- C45 mg too high, 2.5 mg correct
- C5main risk is serious bleeding
- C6cites the current label
- C7patient meets creatinine rule
- workspace factweb fact
Take the task example above. A hospital pharmacist wants to know if a new order of apixaban (a blood thinner) at 5 mg twice a day is right for this patient. The dosing rule is on DailyMed, which says the dose should drop to 2.5 mg twice a day if at least two of three conditions are met: age 80 or older, weight 60 kg or less, and serum creatinine 1.5 mg/dL or higher. We saved an archived copy of this label for grading, but the agent never sees it and has to find the current label on the web. The pharmacist’s workspace contains a PDF and photos with the patient’s details, including age 79, weight 59 kg and creatinine 1.7 mg/dL. The agent must combine the label’s rule with the patient’s details to see that two of the three conditions are met, making 2.5 mg the correct dose, and ignore distractors such as an older lab panel. The task has an atomic rubric listing every fact the answer needs and the evidence behind it. An answer gets full credit only if it meets every item. We used these rubrics to evaluate twelve models from six labs on all 100 tasks.
Model performance
Cost and speed
GPT-6.1 Sol is the most cost-efficient of the four highest-scoring models, at $0.98 per fully correct answer. The corresponding costs are $1.31 for Opus 5.5, $2.06 for Gemini 4 Argon and $5.22 for GPT-6 Astra. GPT-6.1 Sol is also the fastest of the four, with a median attempt of 7.6 minutes. Gemini 4 Argon’s median attempt takes 15.6 minutes and uses about 43,600 output tokens. Time and token figures cover all twelve models.
Full credit and cost per correct answer
Cost covers all twelve models.
Median attempt time and output length vary widely. Opus 5 has the longest median attempt at 30.1 minutes and uses about 75,000 output tokens on a median attempt. Inkling has the shortest median attempt at 3.4 minutes.
Other search engines
With native search, the agent accesses the web through its own shell. In practice, this mostly means fetching pages with curl. We compared native search with Exa, Parallel and Firecrawl by rerunning Gemini 3.8 Flash, GPT-5.6 Sol and Kimi K3 on the same 50 tasks. During these runs, the agent had access to two commands from the engine: one that searched the web and one that fetched a page. There was no cap on search or fetch calls for each rollout.
Change in full credit with a different search engine
Gemini 3.8 Flashnative search 45.3%
GPT-5.6 Solnative search 57.3%
Kimi K3native search 32.0%
Full credit increases in five of the nine combinations. The largest gains are Kimi K3 with Parallel (+8.7 percentage points) and Kimi K3 with Exa (+8.0 points).
Failure Modes
To understand what went wrong, we grouped tasks by the main mistake shown in failed answers. GPT-5.5 suggested a group using those answers and the judge’s notes; we checked every assignment by hand. Each failed attempt was counted under its task’s group.
Failed attempts by failure mode
The largest group involved applying a rule to the wrong case or using the wrong version of it. Tasks in this group accounted for 35% of failed attempts. Other failures involved missing a connection between sources, leaving part of the request unanswered, misunderstanding how a system worked, or misreading a threshold or calculation.
Finding the right source was only part of the challenge. Tasks whose main failure involved finding a source or connecting it to the files accounted for 23% of failed attempts. In many other cases, the model found a relevant source but still reached the wrong conclusion or left part of the work unfinished. The examples below show the specific connection each answer missed.
Email rules shared across a company’s subdomains
Gemini 3.8 Flash judged the company’s corporate email subdomain using only its own 320 messages per day. But Gmail counts all mail from the same main domain—about 29,720 messages per day here. The model did not combine those streams, so it said the subdomain met the rules even though it lacked a required email-authentication record (DKIM).
- Model
- Gemini 3.8 Flash · scored 90.0%
- Rubric item
- The response should conclude corp.nyotaretail.example falls within Gmail's bulk-sender scope despite its own ~320 messages a day, because Gmail counts all messages from the same primary domain — nyotaretail.example's streams total about 29,720 a day — and is therefore non-compliant with the bulk requirement for both SPF and DKIM, since no DKIM selector exists for the stream.
- From the answer
- - corp.nyotaretail.example: COMPLIANT under current low-volume non-bulk sender rules (SPF is valid) [E1, E3, E7, E10], but presents severe operational risk due to lack of DKIM and coupling with organizational DMARC policy [E3].
- Judge’s note
- Contradicts the required conclusion by treating corp as non-bulk and compliant based on its individual volume. It does not apply Gmail's primary-domain aggregation, calculate the combined volume, or classify the missing DKIM as a bulk-requirement failure.
Two copies of the same JavaScript library
The project had two copies of the JavaScript library lodash. Kimi K3 did not explain which copy the application actually loaded: the older, top-level version, rather than the newer copy used by a development tool. That connection matters because a safe copy elsewhere in the project does not make the application’s own copy safe.
- Model
- Kimi K3 · scored 90.0%
- Rubric item
- The response should recognize that a second, nested lodash copy (4.17.21) exists under eslint's devDependency tree but correctly identify it as irrelevant, since src/index.js resolves the top-level lodash (4.17.11) at runtime.
- Judge’s note
- The answer does not identify the second lodash copy as nested under eslint's devDependency tree or explain that it is irrelevant because src/index.js resolves the top-level lodash at runtime. Those explicit requirements are unmet.
A Django app’s startup checks
The fix needed to stop database updates from running in two places whenever the app started, and run them once before deployment instead. Opus 5 discussed removing account setup from the Django startup file but did not explicitly remove the database updates there. Its proposed changes could therefore leave the startup problem in place.
- Model
- Opus 5 · scored 85.0%
- Rubric item
- The response should require database migrations to run only once in a controlled pre-deploy or release step, not from either the Render web-service startCommand or requirements_lab/wsgi.py.
- Judge’s note
- The clause requiring migration execution to be removed from requirements_lab/wsgi.py is not established. Removing the account-seeding block does not explicitly require removing the separate WSGI migration initialization; the rubric requires removal from both startup locations.
Pull requests in GitHub’s issue list
The task asked which changes were essential before releasing an app that used GitHub’s API. Sonnet 5 missed that GitHub’s list of issues can also include pull requests. It did not connect that behavior to the app’s results or explain that filtering out pull requests was an optional improvement, rather than a required fix before release.
- Model
- Sonnet 5 · scored 85.7%
- Rubric item
- The response should recognize that GitHub's Issues endpoint may return both issues and pull requests, and should classify any pull-request filtering recommendation as (b) a recommended improvement that can be deferred rather than (a) a required compatibility change before release.
- Judge’s note
- The response never recognizes that the Issues endpoint may return both issues and pull requests. It also provides no classification of pull-request filtering as a deferrable improvement. Listing separate issues and pull-request endpoints does not establish the required mixed-response behavior.
Reading a traffic-signal guideline
Sonnet 5 found the traffic-signal guideline but read its chart incorrectly. At the busiest measured hour, the side road carried 119 vehicles per hour, below the chart’s required level of roughly 125–135. The model placed the point above the curve and said the four-hour traffic-volume test was likely met, reversing the correct conclusion.
- Model
- Sonnet 5 · scored 90.0%
- Rubric item
- The response should conclude Warrant 2 (Four-Hour Vehicular Volume) is not met. The scored core is the correct conclusion plus any reasonable basis against the Figure 4C-2 (70% factor) curve for 2+ major lanes / 1 minor lane: plotting or discussing the peak point (471 major, 119 minor) below the curve, an approximate curve reading (the curve at the 471 vph peak sits at roughly 125–135 vph on the minor approach; any reading in the 120–140 band supports the conclusion, with the 119 observed below it by about 8 to 15 vph), or a qualitative statement that none of the four highest hours reaches the curve. (The 119 vph peak clears the figure’s 60 vph one-lane floor, but the curve itself governs; a reading near 180 belongs to the 100% Figure 4C-1 curve, not 4C-2.)
- From the answer
- | 2 - Four-Hour Volume | Figure 4C-2 (70% factor) curve, '2+ major-lanes & 1 minor-lane' (asymptote 60vph) [E9] | Minor volumes 96-119 vph plot above curve across the day (visual read) [E9] | Likely YES (visual read; caveat below) |
- Judge’s note
- The required conclusion that Warrant 2 is not met is contradicted. The answer instead reports likely satisfaction and incorrectly places the observations above the applicable curve.
How we built CrossSearch
Each task was written by a screened professional in the task’s field, who had to pass an assessment before submitting. Each professional was limited to at most three tasks for the benchmark. Authors reported spending five to ten hours on each one. The author prepared (1) the request, (2) the starting workspace of files, (3) an archive of the public sources, and (4) a reference answer with a rubric. Before a task was accepted, it had to pass the five checks below.
Review Process
- 1Authorship screenChecked the writing for AI-generated textAutomated
- 2Frontier model rolloutsFour task attempts by a frontier model, read by the task authorAuthor
- 3Rubric alignmentAuthor and judge grades must match after disagreements are resolvedAuthor and LLM judge
- 4Evidence auditCited facts, sources and rubric checkedAutomated
- 5Reviewer decisionAccept, reject or send back for revisionsReviewer
- Accepted
Appendix: Quality & Methods
Task design, file and rubric breakdowns, evidence checks, and evaluation details. Click to expand.
Task collection
Authors were asked to write tasks about work they already knew well. For each task, our collection portal noted whether it was from real work, a reconstruction of a real case, or invented. It also noted the kind of work involved, such as checking a claim, matching local records to a public rule, filling in records with facts from the web, or identifying something before looking it up. In each task, the files had to change what the agent searched for or concluded, and the web had to add something the files couldn’t.
The files come in the formats used in real professional work: photos, scanned pages, spreadsheets, logs and audio recordings. Workspaces contain a median of 3.5 files per task and as many as 27. In 78 tasks, at least one file is something other than plain text, code or CSV. Across all 100 tasks, PDFs appear in 66, tables or spreadsheets (CSV, XLSX) in 60, images in 25, and audio or video in 10. Each task also relies on one to five web sources, 195 across the whole benchmark. Although we archived these sources for grading, the agent never sees that archive and has to find each source on the live web. Rubrics range from 2 to 20 items per task, with a median of 6.
| Format | Tasks out of 100 | Files |
|---|---|---|
| PDFs | 66 | 138 |
| Tables / spreadsheets (CSV, XLSX) | 60 | 88 |
| Text / structured text (TXT, MD, JSON) | 45 | 144 |
| Photos and other image files | 25 | 55 |
| Audio recordings | 9 | 9 |
| Video | 1 | 1 |
Task domains: tasks out of 100 · multiple labels allowed
Rules for files and evidence provenance
To prevent the agent from cheating, file names couldn’t reveal the source or the conclusion, and we instructed experts not to replace a photo, recording or other real file with a note that gave the answer away. Distractors had to be relevant to the case while still being something a careful reader could rule out. Each file came with a record of its origin and usage license, along with the author’s confirmation that it contained no personal information. Every specific claim in the reference answer had to cite its supporting location in the task artifacts, such as a page region in a PDF, a cell range in a spreadsheet, or a time range in a recording or video. An automated check confirmed each cited location at submission, and a human reviewer checked it again before the task was accepted.
Quality gates
Each task went through the five checks above. A task that failed a check went back to its author for revision unless it was rejected outright.
The first check was an authorship screen, where we ran each submitted task through an AI-text detector. A task that scored above 30% went back to the author for rewriting, and tasks that scored 40% or higher were rejected. Afterwards, we ran rollouts on frontier models, in which the task submission portal ran four rollouts on each task, two from GPT-5.6 and two from Claude Opus 4.8 (frontier models at the time), using each provider’s default settings. The task author read through each rollout and checked whether the agent found a valid alternative answer or source, exposed an ambiguity in the question, solved the task without needing both the workspace and the web, or failed for reasons unrelated to what CrossSearch measured. Based on the rollouts, the author then revised the reference answer, the accepted alternatives, and the rubric before submitting a final version for review.
Our automated evidence audit before human review
In the evidence audit, we ran a quality check with a VLM, which inspected each cited region and confirmed that it actually supported the claim that was derived from it. It then continued by checking that each rubric item was necessary to achieve the correct final answer, was supported by the evidence it cited, and matched the prompt originally given to the agent. A second audit then scored the task on eight design dimensions, for example, whether the web source was authentic and missing from the starting workspace, and whether the agent had to work out which rule applied. It listed any defects it found, along with suggested fixes.
Reasons a task could be sent back for further revision
Tasks could be sent back for a variety of reasons, including a source that couldn’t be reached, an inappropriate request, an unsupported reference answer, or imprecise evidence locations, among others. A task could have also been sent back for problems with privacy, licensing or its software environment.
Human review
During rubric alignment, the task author graded each model rollout item by item. An independent model judge (GPT-5.6 at high reasoning effort) graded the same answers without seeing the author’s scores and also evaluated the human-written reference answer. Before submitting a task, authors were instructed to resolve every disagreement by clarifying ambiguous rubric items, fixing the reference answer, accounting for valid alternative answers, or correcting their own grades. This ensured that the rubric was applied consistently to both model and human answers, reducing noise. Both grading passes were then repeated to confirm agreement, and the reference answer had to pass the final rubric.
The final check consisted of a second human reviewer going through a checklist of 12 items that asked them to specifically cover the type of request, the files the agent saw, the reference answer, the rubric, and the frontier rollouts. These 12 items came down to six questions.
- Does the task look like real work, without giving away the answer?
- Are both the files and the web needed, with at least one rubric item that depends on both?
- Can every required source be retrieved (or tied to a fixed date), and is each cited fact actually where the reference says it is?
- Does the rubric grade what the user asked for, and give credit to other valid answers?
- Is the workspace free of grader-only material, and of any metadata that gives the answer away?
- Were the problems found in the frontier rollouts resolved?
Based on the checklist, the reviewer accepted the task, rejected it, or sent it back for revision with a specific reason.
Rubric taxonomy
| Primary rubric group | Items | Share | Item pass rate |
|---|---|---|---|
| Fact / rule identification | 217 | 32.7% | 88.3% |
| Joined application / comparison | 289 | 43.6% | 81.6% |
| Calculation / transformation | 86 | 13.0% | 84.8% |
| Grounding / evidence limits | 71 | 10.7% | 76.7% |
We grouped each rubric item by what a correct answer had to do. Items that required a calculation or conversion went in “Calculation / transformation.” Items that checked citations or whether evidence supported a claim went in “Grounding / evidence limits.” We then split the remaining items by the sources they needed: “Joined application / comparison” required both the local files and the web, while “Fact / rule identification” required only one. We assigned the groups in this order so that each item was counted once.
Item pass rates pool available rubric-item judgments across all 3,600 native-search attempts.
Human–LLM agreement
We use GPT-6.1 Sol as the LLM judge to grade every leaderboard answer against the task’s rubric. To assess the LLM judge’s variance compared to human raters, we ran an ablation on 43 of the benchmark tasks. Each answer received two independent human grades, and we compared five model judges. On whole-answer pass/fail decisions, GPT-6.1 Sol matched an individual human grade 94.8% of the time. The two humans’ ratings agreed 94.2% of the time.
Agreement with human ratings
- Human vs human94.2%
- GPT-6.1 Sol94.8%
- GPT-5.6 Sol93.0%
- Opus 590.1%
- Gemini 3.7 Flash89.0%
- Kimi K384.9%
Evaluation methodology
The leaderboard covered twelve models, each run with its native search and given three scored attempts on all 100 tasks, for a total of 3,600 scored attempts. The search-engine comparison added another 1,350 attempts from three models on 50 of the tasks. Every attempt ran in a fresh Docker container built with the Harbor framework, with the workspace files mounted at /app/artifacts. All models used the Terminus-2 agent and worked through a single shell, where the model could send keystrokes, read the terminal screen, and signal when it was done with the task at hand. In the earlier evaluation’s readable native-search trajectories from every model except Gemini 4 Argon, agents fetched pages with curl in 90% of attempts and with Python HTTP libraries in 40%. Since every model worked through this same harness, our results reflected the harness as well as the model.
All twelve models accepted images and could use a view_image command to look at an image file. Each model ran at extra-high reasoning effort when the provider supported it and at high effort otherwise, with a temperature of 0.6 where the provider allowed it and the provider’s default temperature otherwise.
For grading, we used GPT-6.1 Sol at a temperature of zero, high reasoning effort and a 16,000-token response limit. The judge received the request, the rubric, the reference answer, and accepted variations and alternate evidence, but not the identity of the model being graded. For each criterion, it returned a pass or fail with a rationale and direct quotes from the answer. A validator required the expected rubric IDs and exact, verbatim evidence. Invalid responses were retained separately and were not counted as grades.
An attempt receives full credit only if it meets every rubric item. Pass@1 is the share of all attempts that receive full credit, and pass^3 is the share of tasks where all three attempts do.
Failed attempts and reruns
Some attempts did not produce a usable result, for example because of a blank answer, an infrastructure error or a failed grading call. In the earlier eleven-model evaluation, we reran or regraded these attempts until every model had three complete attempts on every task. Of the 3,300 scored attempts, 499 were reruns of this kind. We did this to evaluate models in their best light. Valid low scores, including zeros, were included, and all three attempts on each task counted, with no best-of-three selection. Inkling 2 Small added 300 trials. Three context-limit outcomes and three timeouts counted as unsuccessful, with null rubric scores.
Data and resources
The blog’s viewer features 10 selected tasks, with their original prompts, example model responses and grading notes. Our Hugging Face release includes the full 100-task benchmark, 4,650 released model runs, complete task prompts and workspace files, and the agent and judge prompt suite.
Task viewer
Retrieval and joining
Request
I'm reviewing the maintenance run of the instrumentation technician from the previous shift, and it looks like our main breaker feed lugs are running a bit hot. However, on my shift just yesterday I had checked the main breaker and it was okay ... I think his measurements could be off. He sent me shots of the cam he used, so can you confirm that the readings for the lugs are trustworthy given the camera he used by telling me the the minimum target size it can resolve at this distance?
Rubric item missed most often
The response should identify the thermal camera used as the base model FLIR E5
Judge’s note
Although the answer recognizes the E5 marking, it conflates the photographed instrument with the E5 Pro and treats the older non-Pro E5/E5-XT as only an alternative identification. This contradicts identification as the base E5 and triggers the rubric's exclusion of other-model identification.
From the answer
The datasheet he supplied is the FLIR Ex PRO-SERIES sheet, whose E5 Pro column gives Spatial Resolution (IFOV) = 3.7 mrad
Show full answerShow less
We release all 100 tasks, the grading protocol and 4,650 model runs in our Hugging Face release. The runs include trajectories, answers, scores, judge decisions and available model reasoning.
The full paper will be released soon.
