suite · audited-with-caveats · verified 2026-08-13
Single-cell Omics Arena
A benchmark for evaluating LLM cell-type annotation across scRNA-seq and single-cell multiomics data.
Audited with caveats: 4 field(s) are marked provisional or conflicted. Warnings are shown next to affected values and these claims are excluded from unqualified comparisons.
Benchmark definition
What is counted
- Version
- initial-release
- Total
- 1226 (Benchmark-wide cell-type annotation task total reported in the abstract.)
- Task formats
- unclassified
- Capabilities
ClassificationScientific reasoning
Version history
| Version | Status | Release / as-of | Total | Formal tracks |
|---|
initial-release
single-cell-omics-arena-soar-initial-release-version | current | 2025-11-24 | 1226 (Benchmark-wide cell-type annotation task total reported in the abstract.) | None registered |
Tracks and subsets
| ID | Count | Basis | Partition? | Notes |
|---|
SOAR-RNA cell-type annotation tasks after preprocessing
soar-rna | 1191 | SOAR-RNA benchmark entries are reported by their annotated cell types after preprocessing. | No | — |
Scientific Task Atlas
Scientific task classification
partial for initial-release. Only high-confidence Scientific Tasks explicitly supported by the creator source are mapped.
| Scientific task | Coverage | Count | Mapping | Evidence |
|---|
| Cell-type annotation | explicitly-in-scope | 1226 tasks
Benchmark-wide cell-type annotation task total reported by the creator paper. | official-taxonomy
high confidence | single-cell-omics-arena-soar-automated-task-1-evidence
|
Relationship registry
How works use this benchmark
Partial claims, non-evaluation uses, and third-party summaries stay visible without entering model comparisons.
Normalized evaluations
evaluation
soar-e5d2b3e-rna-zero-shot-cot-use
Normalizedsubset · n=1191
Work: Single-cell Omics Arena repository result snapshot · source version single-cell-omics-arena-soar-repository-result-snapshot-e5d2b3e
- Selection
- formal-subset · All 1,191 records in the pinned SOAR-RNA JSON artifact, iterated in repository order with shuffle disabled.
- Metrics
- R-1, R-2, R-L, MET., B-1, B-2, BLEU
Not reported / unresolved: repeat count; confidence intervals; contamination or decontamination analysis
Official commit-pinned SOAR-RNA two-call zero-shot chain-of-thought result snapshot; the run covers the formal SOAR-RNA subset, not the full SOAR suite.
Evidence
- repository-path: readme.md; soar_benchmark/configs/cell_type_annotation/experiment_soar_rna.py; soar_benchmark/task.py; soar_benchmark/datasets/soar_rna.json at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (SOAR-RNA zero-shot CoT relation, exact models, formal subset scope, metrics, and linked run.)
Supports: /benchmark_version, /relation_type, /status, /model_ids, /scope, /metric_labels, /evaluation_run_ids, /reporting_gaps, /notes
evaluation
soar-e5d2b3e-rna-zero-shot-use
Normalizedsubset · n=1191
Work: Single-cell Omics Arena repository result snapshot · source version single-cell-omics-arena-soar-repository-result-snapshot-e5d2b3e
- Selection
- formal-subset · All 1,191 records in the pinned SOAR-RNA JSON artifact, iterated in repository order with shuffle disabled.
- Metrics
- R-1, R-2, R-L, MET., B-1, B-2, BLEU
Not reported / unresolved: repeat count; confidence intervals; contamination or decontamination analysis
Official commit-pinned SOAR-RNA zero-shot result snapshot; the run covers the formal SOAR-RNA subset, not the full SOAR suite.
Evidence
- repository-path: readme.md; soar_benchmark/configs/cell_type_annotation/experiment_soar_rna.py; soar_benchmark/datasets/soar_rna.json at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (SOAR-RNA zero-shot relation, exact models, formal subset scope, metrics, and linked run.)
Supports: /benchmark_version, /relation_type, /status, /model_ids, /scope, /metric_labels, /evaluation_run_ids, /reporting_gaps, /notes
Creation, training, validation, or model-selection uses
benchmark creation
single-cell-omics-arena-evaluation-of-large-language-m-single-cell-omics-arena-soar-1-use
Non-evaluationunknown
Work: Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · source version single-cell-omics-arena-evaluation-of-large-language-m-pmc-version-1
- Selection
- not applicable
- Models
- Not reported / not applicable
- Metrics
- Not reported / not applicable
- Linked runs
- None
Not reported / unresolved: No benchmark version is reported.; The benchmark release date is not separately reported.; The official repository license is not reported.; A standalone SOAR-MultiOmics aggregate task count is not printed; PBMC and PFC counts are reported separately.; The source does not describe a separately released grader artifact.
AI-assisted double-pass extraction; values are limited to independently supported claims.
Evidence
- section: Introduction
Supports: /relation_type, /benchmark_id
Evaluation registry
Works and run settings
A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.
soar-e5d2b3e-rna-zero-shotvinitial-release
Evaluated models / systems: gpt-4o-2024-05-13, gpt-4o-mini-2024-07-18
Scopesubset · n=1191
Shots0
Turnssingle-turn
System prompt publicYes
Reasoning / effortnone
BrowserNo
InternetNo
DatabasesNo
Code executionNo
ContainerNo
External tools
Token budget1024 maximum output tokens
Time / cost budgetNot reported
Temperature0
SeedNot reported
RepeatsNot reported
Graderautomatic evaluate-library text-overlap metrics · human review: not reported
Statisticsmetrics computed over all predictions and normalized references
ContaminationNot reported
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|
| R-1 | absolute | source-reported score | all predictions and normalized references | Not reported |
| R-2 | absolute | source-reported score | all predictions and normalized references | Not reported |
| R-L | absolute | source-reported score | all predictions and normalized references | Not reported |
| MET. | absolute | source-reported score | all predictions and normalized references | Not reported |
| B-1 | absolute | source-reported score | all predictions and normalized references; maximum order two | Not reported |
| B-2 | absolute | source-reported score | all predictions and normalized references; maximum order two | Not reported |
| BLEU | absolute | source-reported score | geometric average through order two over all predictions and normalized references | Not reported |
Results
Evidence
- repository-path: soar_benchmark/datasets/soar_rna.json; soar_benchmark/configs/cell_type_annotation/experiment_soar_rna.py; soar_benchmark/task.py; soar_benchmark/pipeline.py; soar_benchmark/prompt_templates/factory.py; analysis/cell_type_annotation/eval_multiple.py at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (Scope, exact models, zero-shot prompt, model call, budgets, tools, and automatic metric implementation.) — supports /benchmark_version, /scope, /model_ids, /protocol, /metrics
- repository-path: readme.md, SOAR-RNA Benchmark, Zero-shot Cell Type Annotation table at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (Exact GPT-4o and GPT-4o mini values for the seven printed metrics.) — supports /results
soar-e5d2b3e-rna-zero-shot-cotvinitial-release
Evaluated models / systems: gpt-4o-2024-05-13, gpt-4o-mini-2024-07-18
Scopesubset · n=1191
Shots0
Turnstwo model calls
System prompt publicYes
Reasoning / effortzero-shot chain-of-thought
BrowserNo
InternetNo
DatabasesNo
Code executionNo
ContainerNo
External tools
Token budget1024 maximum output tokens per call; two calls per example
Time / cost budgetNot reported
Temperature0
SeedNot reported
RepeatsNot reported
Graderautomatic evaluate-library text-overlap metrics · human review: not reported
Statisticsmetrics computed over all predictions and normalized references
ContaminationNot reported
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|
| R-1 | absolute | source-reported score | all predictions and normalized references | Not reported |
| R-2 | absolute | source-reported score | all predictions and normalized references | Not reported |
| R-L | absolute | source-reported score | all predictions and normalized references | Not reported |
| MET. | absolute | source-reported score | all predictions and normalized references | Not reported |
| B-1 | absolute | source-reported score | all predictions and normalized references; maximum order two | Not reported |
| B-2 | absolute | source-reported score | all predictions and normalized references; maximum order two | Not reported |
| BLEU | absolute | source-reported score | geometric average through order two over all predictions and normalized references | Not reported |
Results
Evidence
- repository-path: soar_benchmark/datasets/soar_rna.json; soar_benchmark/configs/cell_type_annotation/experiment_soar_rna.py; soar_benchmark/task.py; soar_benchmark/pipeline.py; soar_benchmark/prompt_templates/factory.py; analysis/cell_type_annotation/eval_multiple.py at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (Scope, exact models, two-call CoT prompt, model calls, budgets, tools, and automatic metric implementation.) — supports /benchmark_version, /scope, /model_ids, /protocol, /metrics
- repository-path: readme.md, SOAR-RNA Benchmark, Zero-shot Chain-of-thought Cell Type Annotation table at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (Exact GPT-4o and GPT-4o mini values for the seven printed metrics.) — supports /results
Comparable result views
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 52.63 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 58.45 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 27.45 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 32.07 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 52.26 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 58.12 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 41.08 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 45.39 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 45.74 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 62.85 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 23.29 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 42.68 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 32.64 | soar-e5d2b3e-rna-zero-shot |
| gpt-4o-2024-05-13 | 51.79 | soar-e5d2b3e-rna-zero-shot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 51.63 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 57.67 | soar-e5d2b3e-rna-zero-shot-cot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 26.6 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 31.55 | soar-e5d2b3e-rna-zero-shot-cot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 51.17 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 57.34 | soar-e5d2b3e-rna-zero-shot-cot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 40.84 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 45.36 | soar-e5d2b3e-rna-zero-shot-cot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 50.29 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 55.27 | soar-e5d2b3e-rna-zero-shot-cot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 27.89 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 32.15 | soar-e5d2b3e-rna-zero-shot-cot |
Accessible data table
| Model | Value | Comparability group |
|---|
| gpt-4o-mini-2024-07-18 | 37.45 | soar-e5d2b3e-rna-zero-shot-cot |
| gpt-4o-2024-05-13 | 42.15 | soar-e5d2b3e-rna-zero-shot-cot |
Evidence and change history
Source locators remain visible; expand an item to inspect the exact Registry fields it supports.
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Data availability · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Data availability · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Data availability · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Introduction · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Introduction · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Evaluation benchmarks of SOAR-RNA and SOAR-MultiOmics · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Evaluation benchmarks of SOAR-RNA and SOAR-MultiOmics · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Method — prompting strategies and benchmark notation · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Introduction · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · other: Article metadata — author-affiliation mapping · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Evaluation benchmarks of SOAR-RNA and SOAR-MultiOmics · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · other: Crossref bibliographic metadata (Resolved from the canonical paper identifier during intake.) · Supports 2 fields
Open source →
/release_date/versions/0/release_date
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Introduction · Supports 4 fields
Open source →
/task_counts/total/task_counts/basis/versions/0/task_counts/total/versions/0/task_counts/basis
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: SOAR-RNA — Cell-type normalization · Supports 4 fields
Open source →
/task_counts/subsets/versions/0/task_counts/subsets/task_counts/subsets/0/versions/0/task_counts/subsets/0
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Data availability · Supports 3 fields
Open source →
/resources/implementations/access/license
single-cell-omics-arena-soar-official-repository-resource · other: Repository tree at commit e5d2b3e2619cb56fece5fba78fae989a67fd0c13 (The pinned tree contains soar_benchmark/datasets/soar_rna.json and no SOAR-MultiOmics benchmark artifact.) · Supports 3 fields
Open source →
/access/level/access/tasks/access/artifacts
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · other: Article metadata and Introduction · Supports 1 field
Open source →
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · other: Article metadata and Introduction · Supports 2 fields
Open source →
/latest_version/versions/0
Single-cell omics arena: evaluation of large language models for automatic cell-type annotations on single-cell omics data via RNA-seq bridging · section: Introduction · Supports 1 field
Open source →
/scientific_task_classification/entries/0
Unresolved field claims
/release_dateProvisional · medium — The creator source does not report a separate benchmark release date. The machine-readable value uses the version-of-record publication date as a proxy and must not be interpreted as an independently verified artifact release date.
Evidence: single-cell-omics-arena-soar-automated-bibliographic-evidence/versions/0/release_dateProvisional · medium — The initial-release snapshot is an Atlas label and its date uses the creator paper's version-of-record publication date because no formal benchmark version date is reported.
Evidence: single-cell-omics-arena-soar-automated-bibliographic-evidence/access/levelConflicted · high — The creator paper states that the benchmark datasets are publicly available, but the pinned official repository snapshot contains the SOAR-RNA artifact and no SOAR-MultiOmics benchmark artifact. The conservative Atlas value is partially-open pending a versioned MultiOmics release.
Evidence: single-cell-omics-arena-soar-automated-metadata-2-evidence, single-cell-omics-arena-soar-pinned-repository-access-evidence/access/licenseProvisional · high — The double-pass review verified the official resource identity but did not establish a redistributable benchmark license; the value remains null pending source-level license verification.
Evidence: single-cell-omics-arena-soar-automated-resource-evidence
View source-level modification history on GitHub →