track · audited · verified 2026-07-21

LAB-Bench FigQA

Multiple-choice interpretation and multi-element reasoning over scientific figures shown without captions or paper context.

Benchmark definition

What is counted

Version: repository-998a8e0
Total: 226 (questions across public and private splits)
Task formats: multiple choice
Capabilities: Evidence synthesisScientific reasoning
Modalities: TextFigureImage

Version history

Version	Status	Release / as-of	Total	Formal tracks
paper-v3 `lab-bench-figqa-paper-v3`	active	2024-07-17	226 (questions across public and private splits)	None registered
repository-998a8e0 `lab-bench-figqa-repository-998a8e0`	current	2025-09-27	226 (questions across public and private splits)	None registered

Tracks and subsets

ID	Count	Basis	Partition?	Notes
Public split `lab-bench-figqa-public`	181	questions	Exclusive & exhaustive	Released in the official repository and Hugging Face dataset.
Private contamination-monitoring split `lab-bench-figqa-private`	45	questions	Exclusive & exhaustive	IDs are published; question contents are withheld.
Creator open-response study `lab-bench-figqa-creator-open-response`	10	modified questions sampled for the creator open-response study	No	A non-exhaustive derived subset; wording was modified to remove multiple-choice framing.

Scientific Task Atlas

Scientific task classification

complete for repository-998a8e0. Single-purpose formal LAB-Bench track.

Scientific task	Coverage	Count	Mapping	Evidence
Scientific evidence interpretation	explicitly-in-scope	226 questions questions across public and private splits	official-track high confidence	`lab-bench-figqa-evidence-paper` Scientific figure interpretation.

Evaluation registry

Works and run settings

A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.

Evaluation run

lab-bench-figqa-anthropic-sonnet45-system-card

From Claude Sonnet 4.5 System Card

Evaluated models / systems: Claude Opus 4, Claude Opus 4.1, Claude Sonnet 4, Claude Sonnet 4.5

Scopetrack

Shots10

Turnssingle-turn

System prompt publicNot reported

Reasoning / effortNot reported

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

GraderNot reported

Statisticsreported point estimate with plotted error bars

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
LAB-Bench score	absolute	proportion	Not reported	Not reported

Results

Model	Metric	Value	n
Claude Sonnet 4	LAB-Bench score	0.398 proportion Point label transcribed from Figure 9.2.4.4.A; numeric error-bar bounds are not reported.	Not reported
Claude Opus 4	LAB-Bench score	0.508 proportion Point label transcribed from Figure 9.2.4.4.A; numeric error-bar bounds are not reported.	Not reported
Claude Opus 4.1	LAB-Bench score	0.481 proportion Point label transcribed from Figure 9.2.4.4.A; numeric error-bar bounds are not reported.	Not reported
Claude Sonnet 4.5	LAB-Bench score	0.497 proportion Point label transcribed from Figure 9.2.4.4.A; numeric error-bar bounds are not reported.	Not reported

Evidence

section: Section 9.2.4.4, printed pages 132–133 (Four selected tracks, 10-shot prompting, no search/bioinformatics tools, threshold, and unreported scope/repeats/grader.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
figure: Figure 9.2.4.4.A, printed page 133 (Exact point labels. Protocol Sonnet 4/4.5 labels are represented in the Claude for Life Sciences run to avoid duplicate result rows.) — supports /results

Evaluation run

lab-bench-figqa-creator-mcq

From LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), Claude 3 Haiku (claude-3-haiku-20240307), Claude 3 Opus (claude-3-opus-20240229), Gemini 1.5 Pro (gemini-1.5-pro-001), GPT-4 Turbo (LAB-Bench snapshot not reported), GPT-4o (LAB-Bench snapshot not reported)

Scopefull · n=226

Shots0

Turnssingle-turn

System prompt publicYes

Reasoning / effortzero-shot chain-of-thought

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

Repeats3

Graderregex multiple-choice parser with Claude 2 fallback · model: Claude 2 fallback parser · human review: no

Statisticsmean across three runs per model

Contaminationfull pre-release snapshot with 80% later public and 20% private contamination-monitoring splits

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	proportion	correct / all questions; mean over three runs	exact choice
Precision (selective accuracy)	absolute	proportion	correct / attempted questions; mean over three runs	insufficient-information responses excluded from denominator
Coverage	absolute	proportion	attempted / all questions; mean over three runs	insufficient-information responses treated as not attempted

Results

Model	Metric	Value	n
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Accuracy	0.46 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Precision (selective accuracy)	0.54 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Coverage	0.85 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3 Opus (claude-3-opus-20240229)	Accuracy	0.24 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3 Opus (claude-3-opus-20240229)	Precision (selective accuracy)	0.31 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3 Opus (claude-3-opus-20240229)	Coverage	0.78 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Gemini 1.5 Pro (gemini-1.5-pro-001)	Accuracy	0.25 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Gemini 1.5 Pro (gemini-1.5-pro-001)	Precision (selective accuracy)	0.34 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Gemini 1.5 Pro (gemini-1.5-pro-001)	Coverage	0.73 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
GPT-4o (LAB-Bench snapshot not reported)	Accuracy	0.29 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
GPT-4o (LAB-Bench snapshot not reported)	Precision (selective accuracy)	0.3 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
GPT-4o (LAB-Bench snapshot not reported)	Coverage	0.97 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
GPT-4 Turbo (LAB-Bench snapshot not reported)	Accuracy	0.25 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
GPT-4 Turbo (LAB-Bench snapshot not reported)	Precision (selective accuracy)	0.28 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
GPT-4 Turbo (LAB-Bench snapshot not reported)	Coverage	0.91 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3 Haiku (claude-3-haiku-20240307)	Accuracy	0.23 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3 Haiku (claude-3-haiku-20240307)	Precision (selective accuracy)	0.24 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226
Claude 3 Haiku (claude-3-haiku-20240307)	Coverage	0.95 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	226

Evidence

section: Appendix D.1–D.4; Table 1 and Appendix Table 6 (Full scope, zero-shot chain-of-thought prompt, no tools, three runs, exact metrics, parser, and model identities.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
table: Tables 2–4, row FigQA (Accuracy, precision, and coverage. Human row excluded.) — supports /results

Evaluation run

lab-bench-figqa-creator-open-response

From LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), GPT-4o (LAB-Bench snapshot not reported)

Scopesubset · n=10

ShotsNot reported

Turnssingle-turn

System prompt publicYes

Reasoning / effortzero-shot chain-of-thought

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

Graderexpert-biologist manual grading against the ideal answer, reviewed by a second expert · human review: yes

Statisticssingle reported accuracy per model

Contaminationfull pre-release snapshot with 80% later public and 20% private contamination-monitoring splits

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	proportion	correct / sampled modified questions	expert judgment against ideal multiple-choice answer

Results

Model	Metric	Value	n
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Accuracy	0.3 proportion Creator Table 5 open-response study.	10
GPT-4o (LAB-Bench snapshot not reported)	Accuracy	0.3 proportion Creator Table 5 open-response study.	10

Evidence

section: Section 2.4 and Appendix B.2 (Subset construction, modified wording, expert grading, and second review.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
table: Table 5 (lab-bench-figqa model accuracy and question count.) — supports /results

Evaluation run

lab-bench-figqa-crop-tool

From Claude Sonnet 4.6 System Card

Evaluated models / systems: Claude Opus 4.6, Claude Sonnet 4.5, Claude Sonnet 4.6

Scopetrack

ShotsNot reported

Turnssingle-turn

System prompt publicNot reported

Reasoning / effortadaptive thinking at max effort

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolssimple image cropping tool

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

Repeats5

GraderNot reported

Statisticsmean over five runs with 95% CI shown

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
FigQA score	absolute	percent	mean over five runs	Not reported

Results

Model	Metric	Value	n
Claude Sonnet 4.6	FigQA score	77.1 percent 95% CI is plotted but numeric bounds are not reported.	Not reported
Claude Sonnet 4.5	FigQA score	59.3 percent 95% CI is plotted but numeric bounds are not reported.	Not reported
Claude Opus 4.6	FigQA score	78.3 percent 95% CI is plotted but numeric bounds are not reported.	Not reported

Evidence

section: Section 2.17.1, printed page 33 (Adaptive thinking, max effort, simple crop tool, five runs, 95% CI, and unreported n/snapshot/grader.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
section: Section 2.17.1 and Figure 2.17.1.A, printed page 33 (All three crop-tool point estimates.) — supports /results

Evaluation run

lab-bench-figqa-no-tools

From Claude Sonnet 4.6 System Card

Evaluated models / systems: Claude Opus 4.6, Claude Sonnet 4.5, Claude Sonnet 4.6

Scopetrack

ShotsNot reported

Turnssingle-turn

System prompt publicNot reported

Reasoning / effortadaptive thinking at max effort

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

Repeats5

GraderNot reported

Statisticsmean over five runs with 95% CI shown

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
FigQA score	absolute	percent	mean over five runs	Not reported

Results

Model	Metric	Value	n
Claude Sonnet 4.6	FigQA score	58.8 percent 95% CI is plotted but numeric bounds are not reported.	Not reported
Claude Sonnet 4.5	FigQA score	53.4 percent 95% CI is plotted but numeric bounds are not reported.	Not reported
Claude Opus 4.6	FigQA score	58 percent 95% CI is plotted but numeric bounds are not reported.	Not reported

Evidence

section: Section 2.17.1, printed page 33 (Adaptive thinking, max effort, no tools, five runs, 95% CI, and unreported n/snapshot/grader.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
section: Section 2.17.1 and Figure 2.17.1.A, printed page 33 (All three no-tool point estimates.) — supports /results

Comparable result views

LAB-Bench score

lab-bench-figqa-anthropic-sonnet45-system-card · lab-bench-figqa-anthropic-10shot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude Sonnet 4	0.398	lab-bench-figqa-anthropic-10shot-no-tools
Claude Opus 4	0.508	lab-bench-figqa-anthropic-10shot-no-tools
Claude Opus 4.1	0.481	lab-bench-figqa-anthropic-10shot-no-tools
Claude Sonnet 4.5	0.497	lab-bench-figqa-anthropic-10shot-no-tools

Accuracy

lab-bench-figqa-creator-mcq · lab-bench-figqa-paper-v3-zero-shot-cot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.46	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Opus (claude-3-opus-20240229)	0.24	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Gemini 1.5 Pro (gemini-1.5-pro-001)	0.25	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
GPT-4o (LAB-Bench snapshot not reported)	0.29	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
GPT-4 Turbo (LAB-Bench snapshot not reported)	0.25	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Haiku (claude-3-haiku-20240307)	0.23	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools

Precision (selective accuracy)

lab-bench-figqa-creator-mcq · lab-bench-figqa-paper-v3-zero-shot-cot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.54	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Opus (claude-3-opus-20240229)	0.31	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Gemini 1.5 Pro (gemini-1.5-pro-001)	0.34	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
GPT-4o (LAB-Bench snapshot not reported)	0.3	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
GPT-4 Turbo (LAB-Bench snapshot not reported)	0.28	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Haiku (claude-3-haiku-20240307)	0.24	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools

Coverage

lab-bench-figqa-creator-mcq · lab-bench-figqa-paper-v3-zero-shot-cot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.85	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Opus (claude-3-opus-20240229)	0.78	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Gemini 1.5 Pro (gemini-1.5-pro-001)	0.73	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
GPT-4o (LAB-Bench snapshot not reported)	0.97	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
GPT-4 Turbo (LAB-Bench snapshot not reported)	0.91	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Haiku (claude-3-haiku-20240307)	0.95	lab-bench-figqa-paper-v3-zero-shot-cot-no-tools

Accuracy

lab-bench-figqa-creator-open-response · lab-bench-figqa-paper-v3-open-response

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.3	lab-bench-figqa-paper-v3-open-response
GPT-4o (LAB-Bench snapshot not reported)	0.3	lab-bench-figqa-paper-v3-open-response

FigQA score

lab-bench-figqa-crop-tool · lab-bench-figqa-sonnet46-adaptive-max-crop-tool-five-runs

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude Sonnet 4.6	77.1	lab-bench-figqa-sonnet46-adaptive-max-crop-tool-five-runs
Claude Sonnet 4.5	59.3	lab-bench-figqa-sonnet46-adaptive-max-crop-tool-five-runs
Claude Opus 4.6	78.3	lab-bench-figqa-sonnet46-adaptive-max-crop-tool-five-runs

FigQA score

lab-bench-figqa-no-tools · lab-bench-figqa-sonnet46-adaptive-max-no-tools-five-runs

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude Sonnet 4.6	58.8	lab-bench-figqa-sonnet46-adaptive-max-no-tools-five-runs
Claude Sonnet 4.5	53.4	lab-bench-figqa-sonnet46-adaptive-max-no-tools-five-runs
Claude Opus 4.6	58	lab-bench-figqa-sonnet46-adaptive-max-no-tools-five-runs

Evidence and change history

Source locators remain visible; expand an item to inspect the exact Registry fields it supports.

LAB-Bench: Measuring Capabilities of Language Models for Biology Research · section: Table 1; Appendix Tables 2–6; task description for FigQA (Task definition, total count, creator protocol, and result row.) · Supports 17 fields

Open source →

/name
/organizations
/release_date
/kind
/domains
/capabilities
/modalities
/task_counts/total
/task_counts/basis
/task_counts/subsets
/coverage_notes
/access/level
/access/license
/versions/0/task_counts/total
/versions/0/task_counts/basis
/versions/0/task_counts/subsets
/scientific_task_classification/entries/0

lab-bench-figqa-repository-resource · repository-path: FigQA/figqa-v1-splits.json; FigQA/figqa-v1-public.jsonl where present; task.py at 998a8e0a40cf116c80e1b0e7a805ebb5fb9fa838 (Public/private/total 181/45/226.) · Supports 13 fields

Open source →

/latest_version
/task_counts/total
/task_counts/basis
/task_counts/subsets
/access/level
/access/tasks
/access/artifacts
/access/grader
/access/license
/resources
/versions/1/task_counts/total
/versions/1/task_counts/basis
/versions/1/task_counts/subsets

View source-level modification history on GitHub →

Access & reproducibility

Access: Partially open
License: CC-BY-SA-4.0
Grader: The official harness scores exact single-choice outputs and reports accuracy, precision, cove…
Implementations: 1 recorded · official