Atlas/Papers/Work detail

preprint · benchmark creator

SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

LatchBio · 2025-12-26

Relationship layer

Benchmark usage

This table records what the work did with each benchmark before attempting to normalize a run. Partial claims remain visible without being treated as comparable evaluations.

benchmark creation

spatialbench-preprint-creation

Benchmark: SpatialBench · version paper-v2

Selection: not applicable
Models: Not reported / not applicable
Metrics: Not reported / not applicable
Linked runs: None

Creator preprint defining the 146-problem paper-v2 benchmark snapshot.

Evidence

page: arXiv v2 abstract and §§2.1, 3.1–3.4 (Introduces and constructs SpatialBench.)
Supports: /benchmark_version, /relation_type, /status, /scope, /notes

evaluation

spatialbench-preprint-evaluation

Benchmark: SpatialBench · version paper-v2

Selection: not applicable · all 146 paper-v2 evaluations
Models: Claude Opus 4.5, Claude Sonnet 4.5, GPT-5.2, GPT-5.1, Gemini 2.5 Pro, Grok-4, Grok-4.1 (SpatialBench paper label)
Metrics: Accuracy, Steps, Latency, Cost
Linked runs: spatialbench-paper-v2-base, spatialbench-paper-v2-claude-code, spatialbench-paper-v2-latch

Base, Claude Code, and Latch harnesses are normalized as separate runs and comparability groups.

Evidence

table: arXiv v2 Tables 1 and 4; §§2.2, 2.5, 3.5–3.7 (Model and harness results over all 146 evaluations.)
Supports: /benchmark_version, /relation_type, /status, /model_ids, /scope, /metric_labels, /evaluation_run_ids, /notes

Normalized evaluation runs

SpatialBench3 runs

Open benchmark record →

Evaluation run

spatialbench-paper-v2-base

From SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

Evaluated models / systems: Claude Opus 4.5, Claude Sonnet 4.5, Gemini 2.5 Pro, GPT-5.1, GPT-5.2, Grok-4, Grok-4.1 (SpatialBench paper label)

Scopefull · n=146

ShotsNot reported

Turnsmulti-turn agent

System prompt publicNo

Reasoning / effortslightly modified Mini-SWE-Bench base harness

BrowserNot reported

InternetNot reported

DatabasesNot reported

Code executionYes

ContainerYes

External toolsinteractive compute and local workspace

Token budgetNot reported

Time / cost budgetmaximum 100 agent steps; exact timeout not reported

TemperatureNot reported

Seedindependent random seeds

Repeats3

Graderfive deterministic grader families · human review: no

Statisticstwo-stage evaluation-weighted mean with t-distribution 95% confidence intervals over per-evaluation means

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 146 evaluation-level means after three repeats	deterministic task-specific grader
Steps	absolute	steps per evaluation	mean over evaluation-level three-run means	Not reported
Latency	absolute	seconds per evaluation	mean over evaluation-level three-run means	Not reported
Cost	absolute	USD per evaluation	mean over evaluation-level three-run means with missing cost logs excluded	Not reported

Results

Model	Metric	Value	n
Claude Opus 4.5	Accuracy	38.36 percent Table 1	146
Claude Opus 4.5	Steps	2.84 steps per evaluation Table 1	146
Claude Opus 4.5	Latency	123.8 seconds per evaluation Table 1	146
Claude Opus 4.5	Cost	0.143 USD per evaluation Table 1	146
Claude Sonnet 4.5	Accuracy	28.31 percent Table 1	146
Claude Sonnet 4.5	Steps	2.43 steps per evaluation Table 1	146
Claude Sonnet 4.5	Latency	115.6 seconds per evaluation Table 1	146
Claude Sonnet 4.5	Cost	0.081 USD per evaluation Table 1	146
GPT-5.2	Accuracy	34.02 percent Table 1	146
GPT-5.2	Steps	2.1 steps per evaluation Table 1	146
GPT-5.2	Latency	89.2 seconds per evaluation Table 1	146
GPT-5.2	Cost	0.037 USD per evaluation Table 1	146
GPT-5.1	Accuracy	27.4 percent Table 1	146
GPT-5.1	Steps	2.38 steps per evaluation Table 1	146
GPT-5.1	Latency	55.8 seconds per evaluation Table 1	146
GPT-5.1	Cost	0.02 USD per evaluation Table 1	146
Gemini 2.5 Pro	Accuracy	20.09 percent Table 1	146
Gemini 2.5 Pro	Steps	3.61 steps per evaluation Table 1	146
Gemini 2.5 Pro	Latency	193.5 seconds per evaluation Table 1	146
Gemini 2.5 Pro	Cost	0.188 USD per evaluation Table 1	146
Grok-4	Accuracy	22.83 percent Table 1	146
Grok-4	Steps	9.9 steps per evaluation Table 1	146
Grok-4	Latency	173.2 seconds per evaluation Table 1	146
Grok-4	Cost	0.048 USD per evaluation Table 1	146
Grok-4.1 (SpatialBench paper label)	Accuracy	24.66 percent Table 1	146
Grok-4.1 (SpatialBench paper label)	Steps	9.93 steps per evaluation Table 1	146
Grok-4.1 (SpatialBench paper label)	Latency	196.4 seconds per evaluation Table 1	146
Grok-4.1 (SpatialBench paper label)	Cost	0.077 USD per evaluation Table 1	146

Evidence

page: arXiv v2 §§3.2–3.7 and Appendix A.4 (Problem anatomy, deterministic graders, interactive compute, isolation, three repeats, 100-step limit, and two-stage statistics.) — supports /benchmark_version, /scope, /protocol, /metrics
table: arXiv v2 Table 1 (Exact base-harness accuracy, steps, latency, cost, and 95% confidence intervals for seven source labels.) — supports /results

Evaluation run

spatialbench-paper-v2-claude-code

From SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

Evaluated models / systems: Claude Opus 4.5, Claude Sonnet 4.5

Scopefull · n=146

ShotsNot reported

Turnsmulti-turn Claude Code agent

System prompt publicNo

Reasoning / effortClaude Code harness

BrowserNot reported

InternetNot reported

DatabasesNot reported

Code executionYes

ContainerYes

External toolsClaude Code harness tools

Token budgetNot reported

Time / cost budgetfixed harness-specific budget; numeric value not reported

TemperatureNot reported

Seedindependent random seeds

Repeats3

Graderfive deterministic grader families · human review: no

Statisticstwo-stage evaluation-weighted mean with t-distribution 95% confidence intervals

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 146 evaluation-level means after three repeats	deterministic task-specific grader

Results

Model	Metric	Value	n
Claude Opus 4.5	Accuracy	48.1 percent Table 4	146
Claude Sonnet 4.5	Accuracy	45.1 percent Table 4	146

Evidence

page: arXiv v2 §§2.5 and 3.5–3.7; Appendix A.4 (Harness separation, full scope, deterministic graders, three repeats, and statistics.) — supports /benchmark_version, /scope, /protocol, /metrics
table: arXiv v2 Table 4 (Exact labeled Claude Code accuracy and 95% confidence intervals.) — supports /results

Evaluation run

spatialbench-paper-v2-latch

From SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

Evaluated models / systems: Claude Opus 4.5

Scopefull · n=146

ShotsNot reported

Turnsmulti-turn Latch agent

System prompt publicNo

Reasoning / effortLatch agent harness

BrowserNot reported

InternetNot reported

DatabasesNot reported

Code executionYes

ContainerYes

External toolsLatch agent harness tools

Token budgetNot reported

Time / cost budgetfixed harness-specific budget; numeric value not reported

TemperatureNot reported

Seedindependent random seeds

Repeats3

Graderfive deterministic grader families · human review: no

Statisticstwo-stage evaluation-weighted mean with t-distribution 95% confidence intervals

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 146 evaluation-level means after three repeats	deterministic task-specific grader

Results

Model	Metric	Value	n
Claude Opus 4.5	Accuracy	61.7 percent Table 4	146

Evidence

page: arXiv v2 §§2.5 and 3.5–3.7; Appendix A.4 (Harness separation, full scope, deterministic graders, three repeats, and statistics.) — supports /benchmark_version, /scope, /protocol, /metrics
table: arXiv v2 Table 4 (Exact labeled Latch accuracy and 95% confidence interval.) — supports /results