Benchmarks/LAB-Bench/LAB-Bench ProtocolQA

track · audited · verified 2026-07-21

LAB-Bench ProtocolQA

Troubleshoots intentionally modified published biological protocols by selecting steps that would repair the stated outcome.

Benchmark definition

What is counted

Version: repository-998a8e0
Total: 135 (questions across public and private splits)
Task formats: multiple choice
Capabilities: TroubleshootingScientific reasoning
Modalities: Text

Version history

Version	Status	Release / as-of	Total	Formal tracks
paper-v3 `lab-bench-protocolqa-paper-v3`	active	2024-07-17	135 (questions across public and private splits)	None registered
repository-998a8e0 `lab-bench-protocolqa-repository-998a8e0`	current	2025-09-27	135 (questions across public and private splits)	None registered

Tracks and subsets

ID	Count	Basis	Partition?	Notes
Public split `lab-bench-protocolqa-public`	108	questions	Exclusive & exhaustive	Released in the official repository and Hugging Face dataset.
Private contamination-monitoring split `lab-bench-protocolqa-private`	27	questions	Exclusive & exhaustive	IDs are published; question contents are withheld.
Creator open-response study `lab-bench-protocolqa-creator-open-response`	20	modified questions sampled for the creator open-response study	No	A non-exhaustive derived subset; wording was modified to remove multiple-choice framing.

Scientific Task Atlas

Scientific task classification

complete for repository-998a8e0. Single-purpose formal LAB-Bench track.

Scientific task	Coverage	Count	Mapping	Evidence
Experiment and protocol planning	explicitly-in-scope	135 questions questions across public and private splits	official-track high confidence	`lab-bench-protocolqa-evidence-paper` Protocol troubleshooting is classified under protocol planning.

Evaluation registry

Works and run settings

A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.

Evaluation run

lab-bench-protocolqa-anthropic

From Claude for Life Sciences

Evaluated models / systems: Claude Sonnet 4, Claude Sonnet 4.5

Scopetrack

Shots10

Turnssingle-turn

System prompt publicNot reported

Reasoning / effortNot reported

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

GraderNot reported

Statisticsreported point estimate with plotted error bars

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
ProtocolQA score	absolute	proportion	Not reported	Not reported

Results

Model	Metric	Value	n
Claude Sonnet 4.5	ProtocolQA score	0.83 proportion Official release rounds the system-card point label 0.833 to 0.83.	Not reported
Claude Sonnet 4	ProtocolQA score	0.74 proportion Official release rounds the system-card point label 0.741 to 0.74.	Not reported

Evidence

section: Section 9.2.4.4 and Figure 9.2.4.4.A, printed pages 132–133 (10-shot, no search/bioinformatics tools, point estimates with error bars, and unreported n/repeats/grader.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
section: Making Claude a better research partner; footnote 1 (Sonnet 4.5 score 0.83, Sonnet 4 score 0.74, and 10-shot prompting.) — supports /results

Evaluation run

lab-bench-protocolqa-anthropic-sonnet45-system-card

From Claude Sonnet 4.5 System Card

Evaluated models / systems: Claude Opus 4, Claude Opus 4.1

Scopetrack

Shots10

Turnssingle-turn

System prompt publicNot reported

Reasoning / effortNot reported

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

GraderNot reported

Statisticsreported point estimate with plotted error bars

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
LAB-Bench score	absolute	proportion	Not reported	Not reported

Results

Model	Metric	Value	n
Claude Opus 4	LAB-Bench score	0.796 proportion Point label transcribed from Figure 9.2.4.4.A; numeric error-bar bounds are not reported.	Not reported
Claude Opus 4.1	LAB-Bench score	0.833 proportion Point label transcribed from Figure 9.2.4.4.A; numeric error-bar bounds are not reported.	Not reported

Evidence

section: Section 9.2.4.4, printed pages 132–133 (Four selected tracks, 10-shot prompting, no search/bioinformatics tools, threshold, and unreported scope/repeats/grader.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
figure: Figure 9.2.4.4.A, printed page 133 (Exact point labels. Protocol Sonnet 4/4.5 labels are represented in the Claude for Life Sciences run to avoid duplicate result rows.) — supports /results

Evaluation run

lab-bench-protocolqa-creator-mcq

From LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), Claude 3 Haiku (claude-3-haiku-20240307), Claude 3 Opus (claude-3-opus-20240229), Gemini 1.5 Pro (gemini-1.5-pro-001), GPT-4 Turbo (LAB-Bench snapshot not reported), GPT-4o (LAB-Bench snapshot not reported), Meta-Llama-3-70B-Instruct (Anyscale API)

Scopefull · n=135

Shots0

Turnssingle-turn

System prompt publicYes

Reasoning / effortzero-shot chain-of-thought

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

Repeats3

Graderregex multiple-choice parser with Claude 2 fallback · model: Claude 2 fallback parser · human review: no

Statisticsmean across three runs per model

Contaminationfull pre-release snapshot with 80% later public and 20% private contamination-monitoring splits

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	proportion	correct / all questions; mean over three runs	exact choice
Precision (selective accuracy)	absolute	proportion	correct / attempted questions; mean over three runs	insufficient-information responses excluded from denominator
Coverage	absolute	proportion	attempted / all questions; mean over three runs	insufficient-information responses treated as not attempted

Results

Model	Metric	Value	n
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Accuracy	0.48 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Precision (selective accuracy)	0.66 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Coverage	0.73 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3 Opus (claude-3-opus-20240229)	Accuracy	0.52 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3 Opus (claude-3-opus-20240229)	Precision (selective accuracy)	0.62 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3 Opus (claude-3-opus-20240229)	Coverage	0.84 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Gemini 1.5 Pro (gemini-1.5-pro-001)	Accuracy	0.47 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Gemini 1.5 Pro (gemini-1.5-pro-001)	Precision (selective accuracy)	0.58 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Gemini 1.5 Pro (gemini-1.5-pro-001)	Coverage	0.81 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
GPT-4o (LAB-Bench snapshot not reported)	Accuracy	0.53 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
GPT-4o (LAB-Bench snapshot not reported)	Precision (selective accuracy)	0.56 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
GPT-4o (LAB-Bench snapshot not reported)	Coverage	0.95 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
GPT-4 Turbo (LAB-Bench snapshot not reported)	Accuracy	0.37 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
GPT-4 Turbo (LAB-Bench snapshot not reported)	Precision (selective accuracy)	0.59 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
GPT-4 Turbo (LAB-Bench snapshot not reported)	Coverage	0.62 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3 Haiku (claude-3-haiku-20240307)	Accuracy	0.45 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3 Haiku (claude-3-haiku-20240307)	Precision (selective accuracy)	0.51 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Claude 3 Haiku (claude-3-haiku-20240307)	Coverage	0.87 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Meta-Llama-3-70B-Instruct (Anyscale API)	Accuracy	0.44 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Meta-Llama-3-70B-Instruct (Anyscale API)	Precision (selective accuracy)	0.49 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135
Meta-Llama-3-70B-Instruct (Anyscale API)	Coverage	0.9 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result.	135

Evidence

section: Appendix D.1–D.4; Table 1 and Appendix Table 6 (Full scope, zero-shot chain-of-thought prompt, no tools, three runs, exact metrics, parser, and model identities.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
table: Tables 2–4, row ProtocolQA (Accuracy, precision, and coverage. Human row excluded.) — supports /results

Evaluation run

lab-bench-protocolqa-creator-open-response

From LAB-Bench: Measuring Capabilities of Language Models for Biology Research

Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), GPT-4o (LAB-Bench snapshot not reported)

Scopesubset · n=20

ShotsNot reported

Turnssingle-turn

System prompt publicYes

Reasoning / effortzero-shot chain-of-thought

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

Graderexpert-biologist manual grading against the ideal answer, reviewed by a second expert · human review: yes

Statisticssingle reported accuracy per model

Contaminationfull pre-release snapshot with 80% later public and 20% private contamination-monitoring splits

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	proportion	correct / sampled modified questions	expert judgment against ideal multiple-choice answer

Results

Model	Metric	Value	n
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	Accuracy	0.3 proportion Creator Table 5 open-response study.	20
GPT-4o (LAB-Bench snapshot not reported)	Accuracy	0.2 proportion Creator Table 5 open-response study.	20

Evidence

section: Section 2.4 and Appendix B.2 (Subset construction, modified wording, expert grading, and second review.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
table: Table 5 (lab-bench-protocolqa model accuracy and question count.) — supports /results

Comparable result views

ProtocolQA score

lab-bench-protocolqa-anthropic · lab-bench-protocolqa-anthropic-life-sciences-10shot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude Sonnet 4.5	0.83	lab-bench-protocolqa-anthropic-life-sciences-10shot-no-tools
Claude Sonnet 4	0.74	lab-bench-protocolqa-anthropic-life-sciences-10shot-no-tools

LAB-Bench score

lab-bench-protocolqa-anthropic-sonnet45-system-card · lab-bench-protocolqa-anthropic-10shot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude Opus 4	0.796	lab-bench-protocolqa-anthropic-10shot-no-tools
Claude Opus 4.1	0.833	lab-bench-protocolqa-anthropic-10shot-no-tools

Accuracy

lab-bench-protocolqa-creator-mcq · lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.48	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Opus (claude-3-opus-20240229)	0.52	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Gemini 1.5 Pro (gemini-1.5-pro-001)	0.47	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
GPT-4o (LAB-Bench snapshot not reported)	0.53	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
GPT-4 Turbo (LAB-Bench snapshot not reported)	0.37	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Haiku (claude-3-haiku-20240307)	0.45	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Meta-Llama-3-70B-Instruct (Anyscale API)	0.44	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools

Precision (selective accuracy)

lab-bench-protocolqa-creator-mcq · lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.66	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Opus (claude-3-opus-20240229)	0.62	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Gemini 1.5 Pro (gemini-1.5-pro-001)	0.58	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
GPT-4o (LAB-Bench snapshot not reported)	0.56	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
GPT-4 Turbo (LAB-Bench snapshot not reported)	0.59	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Haiku (claude-3-haiku-20240307)	0.51	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Meta-Llama-3-70B-Instruct (Anyscale API)	0.49	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools

Coverage

lab-bench-protocolqa-creator-mcq · lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.73	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Opus (claude-3-opus-20240229)	0.84	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Gemini 1.5 Pro (gemini-1.5-pro-001)	0.81	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
GPT-4o (LAB-Bench snapshot not reported)	0.95	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
GPT-4 Turbo (LAB-Bench snapshot not reported)	0.62	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Claude 3 Haiku (claude-3-haiku-20240307)	0.87	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools
Meta-Llama-3-70B-Instruct (Anyscale API)	0.9	lab-bench-protocolqa-paper-v3-zero-shot-cot-no-tools

Accuracy

lab-bench-protocolqa-creator-open-response · lab-bench-protocolqa-paper-v3-open-response

CSV ↓

Accessible data table

Model	Value	Comparability group
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620)	0.3	lab-bench-protocolqa-paper-v3-open-response
GPT-4o (LAB-Bench snapshot not reported)	0.2	lab-bench-protocolqa-paper-v3-open-response

Evidence and change history

Source locators remain visible; expand an item to inspect the exact Registry fields it supports.

LAB-Bench: Measuring Capabilities of Language Models for Biology Research · section: Table 1; Appendix Tables 2–6; task description for ProtocolQA (Task definition, total count, creator protocol, and result row.) · Supports 17 fields

Open source →

/name
/organizations
/release_date
/kind
/domains
/capabilities
/modalities
/task_counts/total
/task_counts/basis
/task_counts/subsets
/coverage_notes
/access/level
/access/license
/versions/0/task_counts/total
/versions/0/task_counts/basis
/versions/0/task_counts/subsets
/scientific_task_classification/entries/0

lab-bench-protocolqa-repository-resource · repository-path: ProtocolQA/protocolqa-v1-splits.json; ProtocolQA/protocolqa-v1-public.jsonl where present; task.py at 998a8e0a40cf116c80e1b0e7a805ebb5fb9fa838 (Public/private/total 108/27/135.) · Supports 13 fields

Open source →

/latest_version
/task_counts/total
/task_counts/basis
/task_counts/subsets
/access/level
/access/tasks
/access/artifacts
/access/grader
/access/license
/resources
/versions/1/task_counts/total
/versions/1/task_counts/basis
/versions/1/task_counts/subsets

View source-level modification history on GitHub →

Access & reproducibility

Access: Partially open
License: CC-BY-SA-4.0
Grader: The official harness scores exact single-choice outputs and reports accuracy, precision, cove…
Implementations: 1 recorded · official