Benchmarks/Biology-Instructions/Biology-Instructions siRNA Efficiency Prediction

track · audited · verified 2026-07-21

Biology-Instructions siRNA Efficiency Prediction

Regression of siRNA efficiency from paired sequence context, evaluated with the SAIS-inspired mixed score.

Benchmark definition

What is counted

Version: emnlp-2025
Total: 66987 (distinct examples across the published train, validation, and test splits)
Task formats: paired sequence regression
Capabilities: PredictionRegression
Modalities: TextDNA or RNA sequence

Version history

Version	Status	Release / as-of	Total	Formal tracks
emnlp-2025 `bioinstruction-sirna-emnlp-2025`	current	2025-11-04	66987 (distinct examples across the published train, validation, and test splits)	None registered

Tracks and subsets

ID	Count	Basis	Partition?	Notes
Training split `bioinstruction-sirna-train`	53592	examples	Exclusive & exhaustive	Published training split.
Validation split `bioinstruction-sirna-validation`	6707	examples	Exclusive & exhaustive	Published validation split.
Test split `bioinstruction-sirna-test`	6688	examples	Exclusive & exhaustive	Held-out split used for creator-paper Tables 4-7.

Scientific Task Atlas

Scientific task classification

complete for emnlp-2025. Single-purpose formal Biology-Instructions evaluation track.

Scientific task	Coverage	Count	Mapping	Evidence
siRNA efficacy prediction	explicitly-in-scope	66987 examples distinct examples across the published train, validation, and test splits	official-track high confidence	`bioinstruction-sirna-evidence-paper` siRNA efficiency prediction.

Evaluation registry

Works and run settings

A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.

Evaluation run

bioinstruction-sirna-closed-baselines

From Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models

Evaluated models / systems: GPT-4o (Biology-Instructions snapshot not reported), GPT-4o-mini (Biology-Instructions snapshot not reported)

Scopesubset · n=6688

Shots0

Turnssingle-turn

System prompt publicYes

Reasoning / effortThe prompt requests a direct JSON answer and no chain-of-thought.

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNot reported

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

Graderfirst-number extraction followed by the creator mixed-score implementation · human review: no

StatisticsPoint metric over the complete published test split, scaled by 100 and rounded to two decimals; no confidence interval is reported.

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Mixed Score	absolute	percent	Creator mixed score combining capped MAE and range-MAE-weighted binary F1, scaled by 100	Not reported

Results

Model	Metric	Value	n
GPT-4o-mini (Biology-Instructions snapshot not reported)	Mixed Score	30.37 percent siRNA creator-paper result; metric scaled by 100.	6688
GPT-4o (Biology-Instructions snapshot not reported)	Mixed Score	0 percent siRNA creator-paper result; metric scaled by 100.	6688

Evidence

table: Table 2 (siRNA test split); Appendix A.3; Table 8; Table 9 closed-source prompt (Scope, prompt, output parser, grader, scaling, and aggregation.) — supports /scope, /benchmark_version, /model_ids, /protocol, /metrics
table: Table 7 (siRNA column) (All registered model values; literature-SOTA row omitted.) — supports /results

Evaluation run

bioinstruction-sirna-creator-systems

From Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models

Evaluated models / systems: ChatMultiOmics stage 1 + balanced stage 2, ChatMultiOmics stage 1 + stage 2, ChatMultiOmics, ChatMultiOmics stage 2 only

Scopesubset · n=6688

Shots0

Turnssingle-turn

System prompt publicYes

Reasoning / effortPsc requests clear, concise task answers and numeric output for regression tasks.

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNot reported

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

Graderfirst-number extraction followed by the creator mixed-score implementation · human review: no

StatisticsPoint metric over the complete published test split, scaled by 100 and rounded to two decimals; no confidence interval is reported.

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Mixed Score	absolute	percent	Creator mixed score combining capped MAE and range-MAE-weighted binary F1, scaled by 100	Not reported

Results

Model	Metric	Value	n
ChatMultiOmics stage 1 + balanced stage 2	Mixed Score	42.92 percent siRNA creator-paper result; metric scaled by 100.	6688
ChatMultiOmics stage 2 only	Mixed Score	4.25 percent siRNA creator-paper result; metric scaled by 100.	6688
ChatMultiOmics stage 1 + stage 2	Mixed Score	56.31 percent siRNA creator-paper result; metric scaled by 100.	6688
ChatMultiOmics	Mixed Score	56.25 percent siRNA creator-paper result; metric scaled by 100.	6688

Evidence

table: Table 2 (siRNA test split); Appendix A.3; Table 8; Section 4.2 Psc prompt (Scope, prompt, output parser, grader, scaling, and aggregation.) — supports /scope, /benchmark_version, /model_ids, /protocol, /metrics
table: Table 7 (siRNA column) (All registered model values; literature-SOTA row omitted.) — supports /results

Evaluation run

bioinstruction-sirna-open-baselines

From Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models

Evaluated models / systems: Alpaca-7B (Biology-Instructions label), BioMedGPT-LM-7B (Biology-Instructions label), Galactica-1.3B (Biology-Instructions label), GLM-4-9B-Chat (Biology-Instructions label), InstructProtein-1.3B (Biology-Instructions label), Llama-molinst-protein-7B (Mol-Ins), Llama2-7B-Chat (Biology-Instructions label), LLaMA3.1-8B-Instruct (Biology-Instructions label), Qwen2-7B (Biology-Instructions label), Vicuna-v1.5-7B (Biology-Instructions label)

Scopesubset · n=6688

Shots0

Turnssingle-turn

System prompt publicNo

Reasoning / effortThe prompt requests the task-formatted answer and says not to explain or repeat.

BrowserNo

InternetNo

DatabasesNo

Code executionNo

ContainerNot reported

External toolsNo

Token budgetNot reported

Time / cost budgetNot reported

TemperatureNot reported

SeedNot reported

RepeatsNot reported

Graderfirst-number extraction followed by the creator mixed-score implementation · human review: no

StatisticsPoint metric over the complete published test split, scaled by 100 and rounded to two decimals; no confidence interval is reported.

ContaminationNot reported

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Mixed Score	absolute	percent	Creator mixed score combining capped MAE and range-MAE-weighted binary F1, scaled by 100	Not reported

Results

Model	Metric	Value	n
LLaMA3.1-8B-Instruct (Biology-Instructions label)	Mixed Score	32.76 percent siRNA creator-paper result; metric scaled by 100.	6688
Qwen2-7B (Biology-Instructions label)	Mixed Score	33.39 percent siRNA creator-paper result; metric scaled by 100.	6688
Llama2-7B-Chat (Biology-Instructions label)	Mixed Score	17.43 percent siRNA creator-paper result; metric scaled by 100.	6688
Alpaca-7B (Biology-Instructions label)	Mixed Score	19.12 percent siRNA creator-paper result; metric scaled by 100.	6688
GLM-4-9B-Chat (Biology-Instructions label)	Mixed Score	23.33 percent siRNA creator-paper result; metric scaled by 100.	6688
Vicuna-v1.5-7B (Biology-Instructions label)	Mixed Score	14.28 percent siRNA creator-paper result; metric scaled by 100.	6688
Galactica-1.3B (Biology-Instructions label)	Mixed Score	33.55 percent siRNA creator-paper result; metric scaled by 100.	6688
InstructProtein-1.3B (Biology-Instructions label)	Mixed Score	5.58 percent siRNA creator-paper result; metric scaled by 100.	6688
Llama-molinst-protein-7B (Mol-Ins)	Mixed Score	13.85 percent siRNA creator-paper result; metric scaled by 100.	6688
BioMedGPT-LM-7B (Biology-Instructions label)	Mixed Score	19.71 percent siRNA creator-paper result; metric scaled by 100.	6688

Evidence

table: Table 2 (siRNA test split); Appendix A.3; Table 8; Table 9 open-source prompt (Scope, prompt, output parser, grader, scaling, and aggregation.) — supports /scope, /benchmark_version, /model_ids, /protocol, /metrics
table: Table 7 (siRNA column) (All registered model values; literature-SOTA row omitted.) — supports /results

Comparable result views

Mixed Score

bioinstruction-sirna-closed-baselines · bioinstruction-sirna-closed-baselines-emnlp-2025

CSV ↓

Accessible data table

Model	Value	Comparability group
GPT-4o-mini (Biology-Instructions snapshot not reported)	30.37	bioinstruction-sirna-closed-baselines-emnlp-2025
GPT-4o (Biology-Instructions snapshot not reported)	0	bioinstruction-sirna-closed-baselines-emnlp-2025

Mixed Score

bioinstruction-sirna-creator-systems · bioinstruction-sirna-creator-systems-emnlp-2025

CSV ↓

Accessible data table

Model	Value	Comparability group
ChatMultiOmics stage 1 + balanced stage 2	42.92	bioinstruction-sirna-creator-systems-emnlp-2025
ChatMultiOmics stage 2 only	4.25	bioinstruction-sirna-creator-systems-emnlp-2025
ChatMultiOmics stage 1 + stage 2	56.31	bioinstruction-sirna-creator-systems-emnlp-2025
ChatMultiOmics	56.25	bioinstruction-sirna-creator-systems-emnlp-2025

Mixed Score

bioinstruction-sirna-open-baselines · bioinstruction-sirna-open-baselines-emnlp-2025

CSV ↓

Accessible data table

Model	Value	Comparability group
LLaMA3.1-8B-Instruct (Biology-Instructions label)	32.76	bioinstruction-sirna-open-baselines-emnlp-2025
Qwen2-7B (Biology-Instructions label)	33.39	bioinstruction-sirna-open-baselines-emnlp-2025
Llama2-7B-Chat (Biology-Instructions label)	17.43	bioinstruction-sirna-open-baselines-emnlp-2025
Alpaca-7B (Biology-Instructions label)	19.12	bioinstruction-sirna-open-baselines-emnlp-2025
GLM-4-9B-Chat (Biology-Instructions label)	23.33	bioinstruction-sirna-open-baselines-emnlp-2025
Vicuna-v1.5-7B (Biology-Instructions label)	14.28	bioinstruction-sirna-open-baselines-emnlp-2025
Galactica-1.3B (Biology-Instructions label)	33.55	bioinstruction-sirna-open-baselines-emnlp-2025
InstructProtein-1.3B (Biology-Instructions label)	5.58	bioinstruction-sirna-open-baselines-emnlp-2025
Llama-molinst-protein-7B (Mol-Ins)	13.85	bioinstruction-sirna-open-baselines-emnlp-2025
BioMedGPT-LM-7B (Biology-Instructions label)	19.71	bioinstruction-sirna-open-baselines-emnlp-2025

Evidence and change history

Source locators remain visible; expand an item to inspect the exact Registry fields it supports.

Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models · table: Table 2 (siRNA row); Appendix A.2-A.3; Table 8; Tables 4-7 (Task definition, split counts, input/output format, metric, and creator evaluation.) · Supports 29 fields

Open source →

/name
/aliases
/summary
/kind
/parent_id
/organizations
/release_date
/latest_version
/domains
/capabilities
/modalities
/task_formats
/task_counts/total
/task_counts/basis
/task_counts/subsets
/access/level
/access/tasks
/access/artifacts
/access/grader
/access/license
/access/biosafety_notes
/resources
/implementations
/versions/0/release_date
/versions/0/as_of
/versions/0/task_counts/total
/versions/0/task_counts/basis
/versions/0/task_counts/subsets
/scientific_task_classification/entries/0

bioinstruction-sirna-repository-resource · repository-path: evaluation/evaluate.py and evaluation/register_tasks.json at 600acaa08c0302e8f5ce86de0fe041f21c13b53e (Public grader implementation, partial artifact release, and absent repository license.) · Supports 7 fields

Open source →

/access/level
/access/tasks
/access/artifacts
/access/grader
/access/license
/resources
/implementations

View source-level modification history on GitHub →

Access & reproducibility

Access: Partially open
License: Not reported
Grader: first-number parser followed by the creator mixed score combining MAE, range MAE, and binary F1.
Implementations: 1 recorded · official

Access details

Tasks: The final paper publishes this task's source, definition, and split counts; the creator repository does not include the complete evaluation inputs and labels.
Artifacts: The official post-processing evaluator is public, but the published test examples, labels, and model outputs are not included in the official repository.
Grader: first-number parser followed by the creator mixed score combining MAE, range MAE, and binary F1.
Biosafety: BioBench Atlas stores task metadata and aggregate results only and does not mirror the upstream biological sequences.