paper · benchmark creator

BLADE: Benchmarking Language Model Agents for Data-Driven Science

University of Washington · UC Berkeley · New York University · Stanford University · University of British Columbia · Microsoft · George Washington University · 2024-11-16

Relationship layer

Benchmark usage

This table records what the work did with each benchmark before attempting to normalize a run. Partial claims remain visible without being treated as comparable evaluations.

No BenchmarkUse relation is normalized for this legacy work yet. Existing EvaluationRuns remain available below.

Normalized evaluation runs

BLADE Decision-Discrimination MCQ1 run

Open benchmark record →

Evaluation run

blade-creator-decision-mcq

From BLADE: Benchmarking Language Model Agents for Data-Driven Science

Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620; BLADE), CodeLlama Instruct 7B (BLADE), DeepSeek-Coder Instruct 6.7B (BLADE), Gemini 1.5 Pro (BLADE snapshot not reported), GPT-3.5 Turbo (BLADE snapshot not reported), GPT-4o (BLADE snapshot not reported), Llama 3 70B (BLADE snapshot not reported), Llama 3 8B (BLADE snapshot not reported), Mixtral 8x22B (BLADE snapshot not reported)

Scopefull · n=188

Shotszero-shot

Turnssingle-turn

System prompt publicYes

Reasoning / effortdirect multiple-choice completion

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsnone

Token budgetNot reported

Time / cost budgetNot reported

Temperature0

SeedNot reported

Repeats1

Graderdeterministic exact option match · human review: no

Statisticsquestion-weighted accuracy with 95% confidence intervals

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Accuracy	absolute	percent	correct choices divided by all 188 MCQs	exact option identifier

No numeric result rows are published yet; the verified protocol remains useful.

Evidence

section: arXiv v3 §§4.1 and 6, Figure 3, and Appendix A.6 Figure 19 (Reports 188 MCQs, nine evaluated model families, temperature zero, accuracy with 95% intervals, and the public prompt.) — supports /benchmark_version, /model_ids, /scope, /protocol/temperature, /protocol/statistical, /metrics, /comparability_group
repository-path: run_mcq.py, run_scripts/sh_run_mcq.sh, blade_bench/baselines/lm/mcq.py, blade_bench/eval/datamodel/run_mcq.py, and blade_bench/conf/llm_config.yml at commit 6118fa8d5007b91aa8c91c518182db82446a4547 (Confirms one direct prompt per item, no tool loop, exact-choice grading, complete public prompt, 20/168 question components, and available exact model strings.) — supports /scope, /protocol/shots, /protocol/turns, /protocol/system_prompt_public, /protocol/reasoning, /protocol/tools, /protocol/repeats, /protocol/grader, /model_ids

BLADE End-to-End Analysis Generation2 runs

Open benchmark record →

Evaluation run

blade-creator-paper

From BLADE: Benchmarking Language Model Agents for Data-Driven Science

Scopefull · n=12

Shotsone-shot

Turnssingle-turn

System prompt publicYes

Reasoning / effortdirect prompted generation without an agent loop

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsnone

Token budgetNot reported

Time / cost budgetNot reported

Temperature0.8

SeedNot reported

Repeats40

Graderhybrid automatic executable and decision matching · model: GPT-4o (exact snapshot not reported) · human review: no

StatisticsFor each dataset and decision type, average precision across all runs and coverage@10 feed a harmonic-mean F1; decision-type F1 values are weighted by ground-truth decision counts. Reported F1 is the mean of 1,000 bootstrap resamples with a percentile-derived 95% interval.

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Average precision	absolute	percent	mean of per-run precision followed by macro averaging across datasets for each decision type	GPT-4o semantic matching plus transformation value/graph matching
Coverage@10	absolute	percent	ground-truth coverage of the union of ten sampled runs followed by macro averaging across datasets for each decision type	run errors count as zero-coverage generations
Decision-type weighted F1-score	absolute	percent	weighted mean of per-decision-type harmonic means of average precision and coverage@10	1000-run bootstrap mean and 95% interval

Results

Model	Metric	Value	n
CodeLlama Instruct 7B (BLADE)	Decision-type weighted F1-score	16.8 percent Forty one-turn generations per dataset; Table 2.	12
DeepSeek-Coder Instruct 6.7B (BLADE)	Decision-type weighted F1-score	33.9 percent Forty one-turn generations per dataset; Table 2.	12
Llama 3 8B (BLADE snapshot not reported)	Decision-type weighted F1-score	29.6 percent Forty one-turn generations per dataset; exact served snapshot not reported.	12
Llama 3 70B (BLADE snapshot not reported)	Decision-type weighted F1-score	36.3 percent Forty one-turn generations per dataset; exact served snapshot not reported.	12
Mixtral 8x22B (BLADE snapshot not reported)	Decision-type weighted F1-score	40.1 percent Forty one-turn generations per dataset; exact served snapshot not reported.	12
GPT-3.5 Turbo (BLADE snapshot not reported)	Decision-type weighted F1-score	30.5 percent Forty one-turn generations per dataset; dated endpoint snapshot not reported.	12
GPT-4o (BLADE snapshot not reported)	Decision-type weighted F1-score	41.7 percent Forty one-turn generations per dataset; dated endpoint snapshot not reported.	12
Gemini 1.5 Pro (BLADE snapshot not reported)	Decision-type weighted F1-score	41.1 percent Forty one-turn generations per dataset; immutable endpoint snapshot not reported.	12
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620; BLADE)	Decision-type weighted F1-score	43.9 percent Forty one-turn generations per dataset; Table 2.	12

Evidence

section: arXiv v3 §§4.2, 5–6 and Appendices A.6–A.7 (Defines the three submitted artifacts, one-shot one-turn prompt, temperature 0.8, 40 runs, GPT-4o-assisted evaluation, average precision, coverage@10, weighted F1, and 1,000-iteration bootstrap.) — supports /benchmark_version, /scope, /protocol, /metrics, /comparability_group
table: Table 2, One-turn Setting (Prints all nine decision-type weighted F1 point estimates and 95% confidence intervals.) — supports /model_ids, /results
repository-path: run_scripts/sh_run_one_turn.sh, blade_bench/baselines/run.py, blade_bench/baselines/lm/analysis.py, blade_bench/conf/llm_config.yml, and blade_bench/eval/ at commit 6118fa8d5007b91aa8c91c518182db82446a4547 (Confirms all 12 datasets, 40 runs, direct generation, public prompt/evaluator, and the available exact model strings.) — supports /model_ids, /protocol/shots, /protocol/turns, /protocol/system_prompt_public, /protocol/tools, /protocol/repeats, /protocol/grader

Evaluation run

blade-creator-react

From BLADE: Benchmarking Language Model Agents for Data-Driven Science

Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620; BLADE), Gemini 1.5 Pro (BLADE snapshot not reported), GPT-3.5 Turbo (BLADE snapshot not reported), GPT-4o (BLADE snapshot not reported), Mixtral 8x22B (BLADE snapshot not reported)

Scopefull · n=12

Shotsone-shot ReAct trajectory

Turnsmulti-turn

System prompt publicYes

Reasoning / effortReAct loop with full prior thought, action, and observation context

BrowserNo

InternetNo

DatabasesNo

Code executionYes

ContainerYes

External toolscomputational notebook cell execution

Token budgetNot reported

Time / cost budgetmaximum 10 agent steps

Temperature0.8

SeedNot reported

Repeats20

Graderhybrid automatic executable and decision matching · model: GPT-4o (exact snapshot not reported) · human review: no

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Average precision	absolute	percent	mean of per-run precision followed by macro averaging across datasets for each decision type	GPT-4o semantic matching plus transformation value/graph matching
Coverage@10	absolute	percent	ground-truth coverage of the union of ten sampled runs followed by macro averaging across datasets for each decision type	run errors count as zero-coverage generations
Decision-type weighted F1-score	absolute	percent	weighted mean of per-decision-type harmonic means of average precision and coverage@10	1000-run bootstrap mean and 95% interval

Results

Model	Metric	Value	n
Mixtral 8x22B (BLADE snapshot not reported)	Decision-type weighted F1-score	40.8 percent Twenty ReAct trajectories per dataset; exact served snapshot not reported.	12
GPT-3.5 Turbo (BLADE snapshot not reported)	Decision-type weighted F1-score	37.2 percent Twenty ReAct trajectories per dataset; dated endpoint snapshot not reported.	12
GPT-4o (BLADE snapshot not reported)	Decision-type weighted F1-score	44.8 percent Twenty ReAct trajectories per dataset; dated endpoint snapshot not reported.	12
Gemini 1.5 Pro (BLADE snapshot not reported)	Decision-type weighted F1-score	40.1 percent Twenty ReAct trajectories per dataset; immutable endpoint snapshot not reported.	12
Claude 3.5 Sonnet (claude-3-5-sonnet-20240620; BLADE)	Decision-type weighted F1-score	43.1 percent Twenty ReAct trajectories per dataset; Table 2.	12

Evidence

section: arXiv v3 §§5–6 and Appendices A.5–A.7 (Defines the ReAct notebook, one trajectory example, Python 3.10 environment, ten steps, temperature 0.8, 20 runs, GPT-4o-assisted evaluation, metrics, and 1,000-iteration bootstrap.) — supports /benchmark_version, /scope, /protocol, /metrics, /comparability_group
table: Table 2, Agent Setting (Prints all five ReAct decision-type weighted F1 point estimates and 95% confidence intervals.) — supports /model_ids, /results
repository-path: run_scripts/sh_run_agent.sh, blade_bench/baselines/agent/react_agent.py, blade_bench/baselines/run.py, blade_bench/conf/llm_config.yml, and blade_bench/eval/ at commit 6118fa8d5007b91aa8c91c518182db82446a4547 (Confirms all 12 datasets, 20 runs, ten-step ReAct execution, public prompt/evaluator, and available exact model strings.) — supports /model_ids, /protocol/shots, /protocol/turns, /protocol/system_prompt_public, /protocol/reasoning, /protocol/tools, /protocol/time_budget, /protocol/repeats, /protocol/grader