official-release · benchmark creator

SpatialBench 159-evaluation repository snapshot

LatchBio · 2026-06-10

Relationship layer

Benchmark usage

This table records what the work did with each benchmark before attempting to normalize a run. Partial claims remain visible without being treated as comparable evaluations.

benchmark creation

spatialbench-repository-creation

Benchmark: SpatialBench · version repo-159-5042c4f

Selection: not applicable
Models: Not reported / not applicable
Metrics: Not reported / not applicable
Linked runs: None

Creator-maintained revised 159-evaluation benchmark snapshot fixed to the registered commit.

Evidence

repository-path: README.md and CHANGELOG.md at commit 5042c4f3ee597da1590650c7b894d068ae968e26 (Defines the revised 159-evaluation version.)
Supports: /benchmark_version, /relation_type, /status, /scope, /notes

evaluation

spatialbench-repository-evaluation

Benchmark: SpatialBench · version repo-159-5042c4f

Selection: not applicable · all 159 evaluations
Models: GPT-5.5, GPT-5.4, Gemini 3.5 Flash, Claude Opus 4.8, Claude Opus 4.6, Claude Opus 4.7, gemini-3.1-pro-preview, GPT-5.2, grok-4.20-beta-0309-reasoning, Claude Sonnet 4.6, Claude Opus 4.5, Claude Sonnet 4.5, GPT-5.1, grok-4-1-fast-reasoning, Grok-4, Gemini 2.5 Pro
Metrics: Accuracy, Cost, Duration
Linked runs: spatialbench-repo-159-mini-swe-agent, spatialbench-repo-159-claude-code, spatialbench-repo-159-pi, spatialbench-repo-159-openai-codex

mini-swe-agent, Claude Code, Pi, and OpenAI Codex are separate normalized runs and comparability groups.

Evidence

repository-path: METHODS.md and results/model_results.csv at commit 5042c4f3ee597da1590650c7b894d068ae968e26 (Full 159-evaluation methods and harness-specific result rows.)
Supports: /benchmark_version, /relation_type, /status, /model_ids, /scope, /metric_labels, /evaluation_run_ids, /notes

Normalized evaluation runs

SpatialBench4 runs

Open benchmark record →

Evaluation run

spatialbench-repo-159-claude-code

From SpatialBench 159-evaluation repository snapshot

Evaluated models / systems: Claude Opus 4.7, Claude Opus 4.8

Scopefull · n=159

ShotsNot reported

Turnsmulti-turn Claude Code

System prompt publicYes

Reasoning / effortmaximum reasoning effort where applicable

BrowserNot reported

InternetYes

DatabasesNot reported

Code executionYes

ContainerYes

External toolsClaude Code with common scientific libraries and network-enabled shell

Token budgetNot reported

Time / cost budget21600 seconds per task with no step limit

TemperatureNot reported

SeedNot reported

Repeats3

Graderfive deterministic grader families · human review: no

Statisticst-distribution 95% confidence interval over per-evaluation means after averaging three runs

Contaminationfull suite withheld to reduce contamination

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 159 evaluation-level means after three repeats	deterministic task-specific grader
Cost	absolute	USD per evaluation	mean over the full result set	Not reported
Duration	absolute	seconds per evaluation	mean over the full result set	Not reported

Results

Model	Metric	Value	n
Claude Opus 4.8	Accuracy	55.35 percent model_results.csv	159
Claude Opus 4.8	Cost	0.8776 USD per evaluation model_results.csv	159
Claude Opus 4.8	Duration	489.16 seconds per evaluation model_results.csv	159
Claude Opus 4.7	Accuracy	51.36 percent model_results.csv	159
Claude Opus 4.7	Cost	0.8023 USD per evaluation model_results.csv	159
Claude Opus 4.7	Duration	532.85 seconds per evaluation model_results.csv	159

Evidence

repository-path: METHODS.md at commit 5042c4f3ee597da1590650c7b894d068ae968e26 (Three runs, max reasoning, container resources, six-hour timeout, network access, binary placement, and aggregation.) — supports /benchmark_version, /scope, /protocol, /metrics
repository-path: results/model_results.csv at commit 5042c4f3ee597da1590650c7b894d068ae968e26; harness=claude-code (Exact model label, accuracy and CI, mean cost, mean duration, and N.) — supports /results

Evaluation run

spatialbench-repo-159-mini-swe-agent

From SpatialBench 159-evaluation repository snapshot

Evaluated models / systems: Claude Opus 4.5, Claude Opus 4.6, Claude Opus 4.7, Claude Opus 4.8, Claude Sonnet 4.5, Claude Sonnet 4.6, Gemini 2.5 Pro, gemini-3.1-pro-preview, Gemini 3.5 Flash, GPT-5.1, GPT-5.2, GPT-5.4, GPT-5.5, Grok-4, grok-4-1-fast-reasoning, grok-4.20-beta-0309-reasoning

Scopefull · n=159

ShotsNot reported

Turnsmulti-turn mini-swe-agent

System prompt publicYes

Reasoning / effortmaximum reasoning effort where applicable

BrowserNot reported

InternetYes

DatabasesNot reported

Code executionYes

ContainerYes

External toolsmini-swe-agent with common scientific libraries and network-enabled shell

Token budgetNot reported

Time / cost budget21600 seconds per task with no step limit

TemperatureNot reported

SeedNot reported

Repeats3

Graderfive deterministic grader families · human review: no

Statisticst-distribution 95% confidence interval over per-evaluation means after averaging three runs

Contaminationfull suite withheld to reduce contamination

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 159 evaluation-level means after three repeats	deterministic task-specific grader
Cost	absolute	USD per evaluation	mean over the full result set	Not reported
Duration	absolute	seconds per evaluation	mean over the full result set	Not reported

Results

Model	Metric	Value	n
GPT-5.5	Accuracy	57.65 percent model_results.csv	159
GPT-5.5	Cost	1.1207 USD per evaluation model_results.csv	159
GPT-5.5	Duration	586.66 seconds per evaluation model_results.csv	159
GPT-5.4	Accuracy	57.44 percent model_results.csv	159
GPT-5.4	Cost	0.577 USD per evaluation model_results.csv	159
GPT-5.4	Duration	1128.75 seconds per evaluation model_results.csv	159
Claude Opus 4.6	Accuracy	52.83 percent model_results.csv	159
Claude Opus 4.6	Cost	0.8456 USD per evaluation model_results.csv	159
Claude Opus 4.6	Duration	609.15 seconds per evaluation model_results.csv	159
Claude Opus 4.8	Accuracy	52.62 percent model_results.csv	159
Claude Opus 4.8	Cost	1.1061 USD per evaluation model_results.csv	159
Claude Opus 4.8	Duration	902.86 seconds per evaluation model_results.csv	159
Claude Opus 4.7	Accuracy	52.41 percent model_results.csv	159
Claude Opus 4.7	Cost	0.9817 USD per evaluation model_results.csv	159
Claude Opus 4.7	Duration	626.84 seconds per evaluation model_results.csv	159
gemini-3.1-pro-preview	Accuracy	51.57 percent model_results.csv	159
gemini-3.1-pro-preview	Cost	0.9362 USD per evaluation model_results.csv	159
gemini-3.1-pro-preview	Duration	1061.63 seconds per evaluation model_results.csv	159
GPT-5.2	Accuracy	50.1 percent model_results.csv	159
GPT-5.2	Cost	0.6024 USD per evaluation model_results.csv	159
GPT-5.2	Duration	931.76 seconds per evaluation model_results.csv	159
Gemini 3.5 Flash	Accuracy	48.85 percent model_results.csv	159
Gemini 3.5 Flash	Cost	2.7608 USD per evaluation model_results.csv	159
Gemini 3.5 Flash	Duration	1145.4 seconds per evaluation model_results.csv	159
grok-4.20-beta-0309-reasoning	Accuracy	45.91 percent model_results.csv	159
grok-4.20-beta-0309-reasoning	Cost	0.1679 USD per evaluation model_results.csv	159
grok-4.20-beta-0309-reasoning	Duration	342.63 seconds per evaluation model_results.csv	159
Claude Sonnet 4.6	Accuracy	44.23 percent model_results.csv	159
Claude Sonnet 4.6	Cost	0.273 USD per evaluation model_results.csv	159
Claude Sonnet 4.6	Duration	405.3 seconds per evaluation model_results.csv	159
Claude Opus 4.5	Accuracy	42.77 percent model_results.csv	159
Claude Opus 4.5	Cost	0.4624 USD per evaluation model_results.csv	159
Claude Opus 4.5	Duration	376.54 seconds per evaluation model_results.csv	159
Claude Sonnet 4.5	Accuracy	41.51 percent model_results.csv	159
Claude Sonnet 4.5	Cost	0.2247 USD per evaluation model_results.csv	159
Claude Sonnet 4.5	Duration	294.44 seconds per evaluation model_results.csv	159
GPT-5.1	Accuracy	39.83 percent model_results.csv	159
GPT-5.1	Cost	0.1574 USD per evaluation model_results.csv	159
GPT-5.1	Duration	309.79 seconds per evaluation model_results.csv	159
grok-4-1-fast-reasoning	Accuracy	33.96 percent model_results.csv	159
grok-4-1-fast-reasoning	Cost	0.0164 USD per evaluation model_results.csv	159
grok-4-1-fast-reasoning	Duration	357 seconds per evaluation model_results.csv	159
Grok-4	Accuracy	31.87 percent model_results.csv	159
Grok-4	Cost	0.4529 USD per evaluation model_results.csv	159
Grok-4	Duration	732.47 seconds per evaluation model_results.csv	159
Gemini 2.5 Pro	Accuracy	28.93 percent model_results.csv	159
Gemini 2.5 Pro	Cost	0.1086 USD per evaluation model_results.csv	159
Gemini 2.5 Pro	Duration	231.07 seconds per evaluation model_results.csv	159

Evidence

repository-path: METHODS.md at commit 5042c4f3ee597da1590650c7b894d068ae968e26 (Three runs, max reasoning, container resources, six-hour timeout, network access, harness placement, and aggregation.) — supports /benchmark_version, /scope, /protocol, /metrics
repository-path: results/model_results.csv at commit 5042c4f3ee597da1590650c7b894d068ae968e26; harness=mini-swe-agent (Exact model label, accuracy and CI, mean cost, mean duration, and N.) — supports /results

Evaluation run

spatialbench-repo-159-openai-codex

From SpatialBench 159-evaluation repository snapshot

Evaluated models / systems: GPT-5.5

Scopefull · n=159

ShotsNot reported

Turnsmulti-turn OpenAI Codex

System prompt publicYes

Reasoning / effortmaximum reasoning effort where applicable

BrowserNot reported

InternetYes

DatabasesNot reported

Code executionYes

ContainerYes

External toolsOpenAI Codex with common scientific libraries and network-enabled shell

Token budgetNot reported

Time / cost budget21600 seconds per task with no step limit

TemperatureNot reported

SeedNot reported

Repeats3

Graderfive deterministic grader families · human review: no

Statisticst-distribution 95% confidence interval over per-evaluation means after averaging three runs

Contaminationfull suite withheld to reduce contamination

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 159 evaluation-level means after three repeats	deterministic task-specific grader
Cost	absolute	USD per evaluation	mean over the full result set	Not reported
Duration	absolute	seconds per evaluation	mean over the full result set	Not reported

Results

Model	Metric	Value	n
GPT-5.5	Accuracy	53.67 percent model_results.csv	159
GPT-5.5	Cost	3.1616 USD per evaluation model_results.csv	159
GPT-5.5	Duration	382.01 seconds per evaluation model_results.csv	159

Evidence

repository-path: METHODS.md at commit 5042c4f3ee597da1590650c7b894d068ae968e26 (Three runs, max reasoning, container resources, six-hour timeout, network access, binary placement, and aggregation.) — supports /benchmark_version, /scope, /protocol, /metrics
repository-path: results/model_results.csv at commit 5042c4f3ee597da1590650c7b894d068ae968e26; harness=openai-codex (Exact model label, accuracy and CI, mean cost, mean duration, and N.) — supports /results

Evaluation run

spatialbench-repo-159-pi

From SpatialBench 159-evaluation repository snapshot

Evaluated models / systems: Gemini 3.5 Flash

Scopefull · n=159

ShotsNot reported

Turnsmulti-turn Pi agent

System prompt publicYes

Reasoning / effortmaximum reasoning effort where applicable

BrowserNot reported

InternetYes

DatabasesNot reported

Code executionYes

ContainerYes

External toolsPi harness with common scientific libraries and network-enabled shell

Token budgetNot reported

Time / cost budget21600 seconds per task with no step limit

TemperatureNot reported

SeedNot reported

Repeats3

Graderfive deterministic grader families · human review: no

Statisticst-distribution 95% confidence interval over per-evaluation means after averaging three runs

Contaminationfull suite withheld to reduce contamination

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	mean over 159 evaluation-level means after three repeats	deterministic task-specific grader
Cost	absolute	USD per evaluation	mean over the full result set	Not reported
Duration	absolute	seconds per evaluation	mean over the full result set	Not reported

Results

Model	Metric	Value	n
Gemini 3.5 Flash	Accuracy	55.56 percent model_results.csv	159
Gemini 3.5 Flash	Cost	1.4254 USD per evaluation model_results.csv	159
Gemini 3.5 Flash	Duration	1739.29 seconds per evaluation model_results.csv	159

Evidence

repository-path: METHODS.md at commit 5042c4f3ee597da1590650c7b894d068ae968e26 (Three runs, max reasoning, container resources, six-hour timeout, network access, and aggregation.) — supports /benchmark_version, /scope, /protocol, /metrics
repository-path: results/model_results.csv at commit 5042c4f3ee597da1590650c7b894d068ae968e26; harness=pi (Exact model label, accuracy and CI, mean cost, mean duration, and N.) — supports /results