Evaluation run
bioinstruction-modification-closed-baselines
Evaluated models / systems: GPT-4o (Biology-Instructions snapshot not reported), GPT-4o-mini (Biology-Instructions snapshot not reported)
Scopesubset · n=1200
Shots0
Turnssingle-turn
System prompt publicYes
Reasoning / effortThe prompt requests a direct JSON answer and no chain-of-thought.
BrowserNo
InternetNo
DatabasesNo
Code executionNo
ContainerNot reported
External toolsNo
Token budgetNot reported
Time / cost budgetNot reported
TemperatureNot reported
SeedNot reported
RepeatsNot reported
Gradermodification-label extraction with sentiment fallback for none, followed by macro ROC AUC · model: cardiffnlp/twitter-roberta-base-sentiment-latest fallback · human review: no
StatisticsPoint metric over the complete published test split, scaled by 100 and rounded to two decimals; no confidence interval is reported.
ContaminationNot reported
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| AUC | absolute | percent | Macro ROC AUC across modification labels on held-out test examples, scaled by 100 | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| GPT-4o-mini (Biology-Instructions snapshot not reported) | AUC | 50.49 percent Modif creator-paper result; metric scaled by 100. | 1200 |
| GPT-4o (Biology-Instructions snapshot not reported) | AUC | 50.47 percent Modif creator-paper result; metric scaled by 100. | 1200 |
Evidence
- table: Table 2 (Modif test split); Appendix A.3; Table 8; Table 9 closed-source prompt (Scope, prompt, output parser, grader, scaling, and aggregation.) — supports /scope, /benchmark_version, /model_ids, /protocol, /metrics
- table: Table 5 (Modif column) (All registered model values; literature-SOTA row omitted.) — supports /results