Evaluation run
lab-bench-litqa2-creator-mcq
From LAB-Bench: Measuring Capabilities of Language Models for Biology Research
Evaluated models / systems: Claude 3.5 Sonnet (claude-3-5-sonnet-20240620), Claude 3 Haiku (claude-3-haiku-20240307), Claude 3 Opus (claude-3-opus-20240229), Gemini 1.5 Pro (gemini-1.5-pro-001), GPT-4 Turbo (LAB-Bench snapshot not reported), GPT-4o (LAB-Bench snapshot not reported), Meta-Llama-3-70B-Instruct (Anyscale API)
Scopefull · n=248
Shots0
Turnssingle-turn
System prompt publicYes
Reasoning / effortzero-shot chain-of-thought
BrowserNo
InternetNo
DatabasesNo
Code executionNo
ContainerNo
External toolsNo
Token budgetNot reported
Time / cost budgetNot reported
TemperatureNot reported
SeedNot reported
Repeats3
Graderregex multiple-choice parser with Claude 2 fallback · model: Claude 2 fallback parser · human review: no
Statisticsmean across three runs per model
Contaminationfull pre-release snapshot with 80% later public and 20% private contamination-monitoring splits
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Accuracy | absolute | proportion | correct / all questions; mean over three runs | exact choice |
| Precision (selective accuracy) | absolute | proportion | correct / attempted questions; mean over three runs | insufficient-information responses excluded from denominator |
| Coverage | absolute | proportion | attempted / all questions; mean over three runs | insufficient-information responses treated as not attempted |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) | Accuracy | 0.06 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) | Precision (selective accuracy) | 0.47 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3.5 Sonnet (claude-3-5-sonnet-20240620) | Coverage | 0.12 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3 Opus (claude-3-opus-20240229) | Accuracy | 0.06 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3 Opus (claude-3-opus-20240229) | Precision (selective accuracy) | 0.43 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3 Opus (claude-3-opus-20240229) | Coverage | 0.14 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Gemini 1.5 Pro (gemini-1.5-pro-001) | Accuracy | 0.1 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Gemini 1.5 Pro (gemini-1.5-pro-001) | Precision (selective accuracy) | 0.44 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Gemini 1.5 Pro (gemini-1.5-pro-001) | Coverage | 0.23 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| GPT-4o (LAB-Bench snapshot not reported) | Accuracy | 0.27 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| GPT-4o (LAB-Bench snapshot not reported) | Precision (selective accuracy) | 0.46 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| GPT-4o (LAB-Bench snapshot not reported) | Coverage | 0.58 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| GPT-4 Turbo (LAB-Bench snapshot not reported) | Accuracy | 0.12 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| GPT-4 Turbo (LAB-Bench snapshot not reported) | Precision (selective accuracy) | 0.4 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| GPT-4 Turbo (LAB-Bench snapshot not reported) | Coverage | 0.31 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3 Haiku (claude-3-haiku-20240307) | Accuracy | 0.27 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3 Haiku (claude-3-haiku-20240307) | Precision (selective accuracy) | 0.38 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Claude 3 Haiku (claude-3-haiku-20240307) | Coverage | 0.7 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Meta-Llama-3-70B-Instruct (Anyscale API) | Accuracy | 0.35 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Meta-Llama-3-70B-Instruct (Anyscale API) | Precision (selective accuracy) | 0.38 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
| Meta-Llama-3-70B-Instruct (Anyscale API) | Coverage | 0.92 proportion Creator-paper full-snapshot result; human-baseline row is not encoded as a model result. | 248 |
Evidence
- section: Appendix D.1–D.4; Table 1 and Appendix Table 6 (Full scope, zero-shot chain-of-thought prompt, no tools, three runs, exact metrics, parser, and model identities.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics
- table: Tables 2–4, row LitQA2 (Accuracy, precision, and coverage. Human row excluded.) — supports /results