suite · audited · verified 2026-07-21

SCIGYM

An agentic systems-biology suite in which language models iteratively perturb simulated SBML systems, analyze time-series observations in Python, and reconstruct hidden biological reactions.

Release, split, and protocol audit: SCIGYM releases 350 SBML systems: 137 small systems with fewer than ten reactions and 213 large systems with up to 400 reactions. Only the small track was evaluated in the creator paper. Its public evaluation artifact proves three episodes for every model–system pair; the code and dataset are public, but upstream states no code or data license.

Benchmark definition

What is counted

Version: 2025 release
Total: 350 (distinct curated BioModels systems released as SBML benchmark instances)
Task formats: interactive simulated experiment; SBML reaction-network reconstruction
Capabilities: Experiment planningData analysisCodingTool useScientific reasoning
Modalities: TextTableCode

Version history

Version	Status	Release / as-of	Total	Formal tracks
2025 release `scigym-2025-release`	current	2025-05-16	350 (distinct curated BioModels systems released as SBML benchmark instances)	scigym-small, scigym-large

Tracks and subsets

ID	Count	Basis	Partition?	Notes
Small systems `scigym-small-systems`	137	systems with fewer than 10 reactions in the evaluated small split	Exclusive & exhaustive	All six creator-paper models were evaluated on this split.
Large systems `scigym-large-systems`	213	remaining released systems with up to 400 reactions in the large split	Exclusive & exhaustive	Released by the creators but not evaluated in the paper.

Registered child tracks

SCIGYM Large

The formally released SCIGYM track containing the 213 systems not included in the creator paper's model evaluation, with systems reaching up to 400 reactions.

0 evaluation run(s)

SCIGYM Small

The formally released and creator-evaluated SCIGYM track containing biological systems with fewer than ten reactions.

2 evaluation run(s)

Scientific Task Atlas

Scientific task classification

complete for 2025 release. All released systems use the same simulator-based reaction-network reconstruction task.

Scientific task	Coverage	Count	Mapping	Evidence
Reaction-network reconstruction	explicitly-in-scope	350 systems distinct curated BioModels systems released as SBML benchmark instances	official-taxonomy high confidence	`scigym-evidence-release-counts` `scigym-evidence-taxonomy` Hidden biological reactions are reconstructed from interventions and trajectories.
Simulation-based experiment	explicitly-in-scope	350 systems distinct curated BioModels systems released as SBML benchmark instances	official-taxonomy high confidence	`scigym-evidence-release-counts` `scigym-evidence-taxonomy` The same systems support iterative in-silico perturbation experiments; overlapping claims are not summed.

Scientific coverage notes

Domain	Coverage	Count	Interpretation
Molecular and cell biology	explicitly-in-scope	Not reported	The source systems include signaling, metabolic, regulatory, and other biological processes, but the creators do not publish a mutually exclusive molecular/cell-biology count.
Genomics	unknown	Not reported	Gene-regulatory networks are named as an example system class, but no official task-level genomics count is published and the benchmark does not use genomic sequence data.
Transcriptomics	not-in-scope	0	The paper motivates the task by analogy to Perturb-seq and spatial transcriptomics, but SCIGYM inputs are SBML systems and simulated concentration trajectories rather than transcriptomic measurements.
Single-cell	not-in-scope	0	No single-cell measurement is an official benchmark input or target.
Protein design	not-in-scope	0	Agents reconstruct reaction networks; they do not design or optimize protein sequences or structures.
Protein-protein binding	not-in-scope	0	Species can represent proteins, but no task evaluates protein-protein binding prediction or affinity.
Protein-ligand binding	not-in-scope	0	No task evaluates protein-ligand binding prediction or affinity.

Evaluation registry

Works and run settings

A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.

Evaluation run

scigym-small-creator-paper

From Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab

Evaluated models / systems: Claude 3.5 Haiku 20241022 (SCIGYM), Claude 3.7 Sonnet 20250219 (SCIGYM), Gemini 2.5 Flash Preview 04-17 (SCIGYM), Gemini 2.5 Pro Preview 03-25 (SCIGYM), GPT-4.1 2025-04-14 (SCIGYM), GPT-4.1 Mini 2025-04-14 (SCIGYM)

Scopefull · n=137

ShotsNot reported

Turnsmulti-turn

System prompt publicYes

Reasoning / effortReAct-style Thoughts–Actions–Observations agent; no provider reasoning-effort control is reported.

BrowserNo

InternetNo

DatabasesNo

Code executionYes

ContainerNot reported

External toolsTellurium, libRoadRunner, libSBML, pandas, numpy, SCIGYM experiment API

Token budgetmaximum 8,192 output tokens per model response

Time / cost budgetmaximum 20 action iterations plus up to 3 invalid-submission debugging iterations

TemperatureNot reported

SeedNot reported

Repeats3

Graderdeterministic SBML structural and simulation evaluator · human review: no

StatisticsTable 1 reports arithmetic means across the small benchmark instances. The public artifact supplies three episodes for every model-system pair; Table 1 does not print confidence intervals.

ContaminationReference models are de-identified by stripping metadata, shuffling components, and replacing component IDs; species names are retained. No model-training decontamination test is reported.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Network Topology Score (NTS) F1	absolute	score	pairwise species-interaction F1 with duplicate relationships counted once, then averaged across systems/episodes	relationship type includes reactant-product, reactant-modifier, and modifier-product
Simulation Trajectory Error (STE)	absolute	SMAPE error	SMAPE averaged across species and benchmark systems/episodes	evaluated under original and perturbed initial conditions
RMS with modifiers — Precision	absolute	score	exact reactant/product/modifier reaction matching averaged across systems/episodes	reactants, products, and modifiers must match
RMS with modifiers — Recall	absolute	score	exact reactant/product/modifier reaction matching averaged across systems/episodes	reactants, products, and modifiers must match
RMS with modifiers — F1	absolute	score	harmonic mean of with-modifier reaction precision and recall averaged across systems/episodes	reactants, products, and modifiers must match
RMS without modifiers — Precision	absolute	score	exact reactant/product reaction matching averaged across systems/episodes	modifiers are ignored
RMS without modifiers — Recall	absolute	score	exact reactant/product reaction matching averaged across systems/episodes	modifiers are ignored
RMS without modifiers — F1	absolute	score	harmonic mean of without-modifier reaction precision and recall averaged across systems/episodes	modifiers are ignored

Results

Model	Metric	Value	n
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	Simulation Trajectory Error (STE)	0.4181 SMAPE error Table 1; three public episodes per system.	137
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	RMS with modifiers — Precision	0.1527 score Table 1; three public episodes per system.	137
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	RMS with modifiers — Recall	0.1071 score Table 1; three public episodes per system.	137
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	RMS with modifiers — F1	0.1217 score Table 1; three public episodes per system.	137
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	RMS without modifiers — Precision	0.2399 score Table 1; three public episodes per system.	137
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	RMS without modifiers — Recall	0.1839 score Table 1; three public episodes per system.	137
Gemini 2.5 Flash Preview 04-17 (SCIGYM)	RMS without modifiers — F1	0.2005 score Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	Simulation Trajectory Error (STE)	0.6007 SMAPE error Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	RMS with modifiers — Precision	0.1516 score Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	RMS with modifiers — Recall	0.1253 score Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	RMS with modifiers — F1	0.132 score Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	RMS without modifiers — Precision	0.253 score Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	RMS without modifiers — Recall	0.2313 score Table 1; three public episodes per system.	137
GPT-4.1 Mini 2025-04-14 (SCIGYM)	RMS without modifiers — F1	0.2322 score Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	Simulation Trajectory Error (STE)	0.6281 SMAPE error Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	RMS with modifiers — Precision	0.0858 score Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	RMS with modifiers — Recall	0.0421 score Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	RMS with modifiers — F1	0.053 score Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	RMS without modifiers — Precision	0.1454 score Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	RMS without modifiers — Recall	0.0805 score Table 1; three public episodes per system.	137
Claude 3.5 Haiku 20241022 (SCIGYM)	RMS without modifiers — F1	0.0987 score Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	Simulation Trajectory Error (STE)	0.3212 SMAPE error Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	RMS with modifiers — Precision	0.2138 score Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	RMS with modifiers — Recall	0.1664 score Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	RMS with modifiers — F1	0.1817 score Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	RMS without modifiers — Precision	0.3781 score Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	RMS without modifiers — Recall	0.3219 score Table 1; three public episodes per system.	137
Gemini 2.5 Pro Preview 03-25 (SCIGYM)	RMS without modifiers — F1	0.3383 score Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	Simulation Trajectory Error (STE)	0.4611 SMAPE error Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	RMS with modifiers — Precision	0.2067 score Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	RMS with modifiers — Recall	0.1597 score Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	RMS with modifiers — F1	0.174 score Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	RMS without modifiers — Precision	0.3517 score Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	RMS without modifiers — Recall	0.2888 score Table 1; three public episodes per system.	137
GPT-4.1 2025-04-14 (SCIGYM)	RMS without modifiers — F1	0.3038 score Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	Simulation Trajectory Error (STE)	0.3615 SMAPE error Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	RMS with modifiers — Precision	0.178 score Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	RMS with modifiers — Recall	0.1698 score Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	RMS with modifiers — F1	0.1688 score Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	RMS without modifiers — Precision	0.316 score Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	RMS without modifiers — Recall	0.317 score Table 1; three public episodes per system.	137
Claude 3.7 Sonnet 20250219 (SCIGYM)	RMS without modifiers — F1	0.3047 score Table 1; three public episodes per system.	137

Evidence

section: NeurIPS paper §§3.2–3.3 and 5; Appendix A (Defines the ReAct loop, three action types, public prompt, initial-concentration experiment, 20 action iterations, three debugging iterations, six exact model versions, full 137-system small scope, and NTS/RMS/STE.) — supports /benchmark_version, /scope, /model_ids, /protocol/shots, /protocol/turns, /protocol/system_prompt_public, /protocol/reasoning, /protocol/tools, /protocol/time_budget, /protocol/grader, /protocol/statistical, /protocol/contamination, /metrics, /comparability_group
dataset-card: data/small-00000-of-00001.parquet at commit 7d472c12855d46702c4915892578290355894c1a (2,466 rows; 137 unique systems, six exact model strings, and exactly three rows for every model-system pair.) — supports /model_ids, /protocol/repeats, /protocol/statistical
repository-path: README.md, scigym/llm.py, scigym/controller.py, scigym/system_prompts/, and scigym/evaluator.py at commit d290bb04bf54aad1c473c4701e0d0d88013c4f91 (Confirms exact API strings, maximum 8,192 response tokens, no browser/network/database tool, public Python/SBML tool loop, and deterministic evaluator.) — supports /model_ids, /protocol/system_prompt_public, /protocol/tools, /protocol/token_budget, /protocol/grader, /metrics
table: NeurIPS paper Table 1 (Prints STE and RMS precision/recall/F1 with and without modifiers for all six models; no confidence bounds are printed.) — supports /results

Evaluation run

scigym-small-zero-shot

From Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab

Scopefull · n=137

Shotszero-shot

Turnsone direct submission, with up to three debugging responses after invalid submissions

System prompt publicYes

Reasoning / effortdirect prompting without experimental feedback

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNo

External toolsnone

Token budgetmaximum 8,192 output tokens per model response

Time / cost budgetone direct submission plus up to 3 invalid-submission debugging iterations

TemperatureNot reported

SeedNot reported

RepeatsNot reported

Graderdeterministic SBML structural and simulation evaluator · human review: no

StatisticsFigure 5 plots per-system agent and zero-shot scores; exact aggregate baseline values and confidence intervals are not tabulated.

ContaminationThe same de-identified small systems are used; no model-training decontamination test is reported.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Network Topology Score (NTS) F1	absolute	score	per-system structural F1	duplicate relationships counted once
Simulation Trajectory Error (STE)	absolute	SMAPE error	species-level SMAPE averaged within system	original and perturbed initial conditions
RMS with modifiers — F1	absolute	score	exact-reaction F1	reactants products and modifiers must match
RMS without modifiers — F1	absolute	score	exact-reaction F1	modifiers ignored

No numeric result rows are published yet; the verified protocol remains useful.