Inclusion boundary
Production exports accept benchmark-creator sources, official model-provider materials, peer-reviewed papers, and stable versioned preprints from arXiv, bioRxiv, medRxiv, or ChemRxiv. Independent papers use independent_reproduction. Ordinary blogs, secondary leaderboards, news summaries, and unstable or unverifiable claims remain excluded.
Five entities, five meanings
- Benchmark defines the scientific task, access, counts, taxonomy, and runnable status.
- Work identifies the versioned paper, system card, report, or official page making a claim.
- BenchmarkUse states how a Work used a Benchmark: creation, evaluation, training, fine-tuning, validation, model selection, or external result summary. It can remain partial when settings are missing.
- EvaluationRun captures one coherent, sufficiently reported scope and protocol within a work.
- Model preserves the exact reported model identity without fuzzy merging.
Scope is explicit
full is accepted only when the realized n matches the benchmark version total. subset requires a subset ID, filter, and realized count. track refers to a formal benchmark track. When the source does not let us determine these, scope is unknown; we do not infer full coverage.
Missing means missing
Unreported values use null plus reporting_status: not_reported. Zero is never used as a missing-value code. The LifeSciBench binding example demonstrates the difference: binding is explicitly in scope, but no standalone count is reported.
Scientific Task is a separate axis
Domain describes scientific context, Capability describes what the evaluated system must do, and Modality describes the input. Scientific Task identifies the actual scientific problem—such as protein monomer structure prediction, protein sequence design, PPI, protein-ligand pose or affinity prediction, promoter detection, RNA design, cell-type annotation, or end-to-end computational analysis.
Mappings use only a creator taxonomy, formal track, or locatable official artifact. We record the most specific supported task and never duplicate its parent on the same record. Mixed suites may remain partial. Parent task pages roll up their leaf mappings, while every count claim retains its original unit and basis.
Metrics and comparability
Metrics retain their source label, unit, direction, range, aggregation, threshold, and tolerance. Result rows only reference a metric declared by their run. Charts compare models only inside an explicit comparability_group, which must agree on benchmark version, scope, metric, and key protocol settings.
A labeled improvement is stored as a delta metric with an explicit baseline model. It is never displayed as an absolute score. Unlabeled bar heights and line positions are not digitized.
Evidence
Each critical field points to an official work plus a locator—page, section, figure, or table—and lists the JSON paths it supports. Remote papers and webpages are linked rather than mirrored unless redistribution permission is explicit.
v1.1 field-level audit
Each benchmark family moves independently from legacy to audited or audited-with-caveats. Audited records carry structured version history, permanent resource and evidence IDs, typed locators, and resolvable field paths. A provisional or conflicted value stays visible beside an explicit warning and remains machine-readable in JSON and CSV.
legacy. CI requires every legacy record to have a matching machine-readable exception in registry metadata.scope: full. Provisional or conflicted result rows remain available for inspection and download but do not enter comparison charts.Release and reproducibility
Verified YAML is validated, normalized once, and emitted as website data, JSON, CSV, and schemas. Monthly release snapshots are maintainer-triggered and immutable. Weekly link checks open issues but never mutate facts.
Local paper intake
Every week, high-precision candidates are discovered through Europe PMC, Crossref, and arXiv. At most ten issues are opened using fixed protein (4), DNA/RNA (2), small-molecule (2), and omics/cell (2) quotas. Exact benchmark aliases, creator citations, official repositories, and explicit bio/chem benchmark language are accepted as discovery signals. Candidate issues never enter the Registry.
Discovery never launches extraction. The repository owner explicitly selects an issue or paper URL before the local Codex workflow runs. Only legal open full text or a submitter-authorized source is read. Sources are capped at 45 MiB and 150 pages, fingerprinted with SHA256, kept in a gitignored local temporary directory, deleted during cleanup, and never committed or redistributed.
Independent local double-pass verification
The extractor and verifier run as two new ephemeral codex exec sessions under read-only sandboxes. They use the locally authenticated Codex CLI rather than a repository API key. Neither pass may use the network, apps, MCP servers, or outside knowledge. The extractor emits Pydantic-schema claims rather than YAML; the verifier independently relocates every page, table, figure, and number. Only claims supported at high confidence by both passes can reach the deterministic Registry generator.
Local Codex opens a Ready PR under the owner's account but never merges it. A paper-intake PR passes paper-owner-gate only after wang422003 comments /approve-paper-intake <full-current-head-sha>. A later push changes the SHA and invalidates the old approval. Regular CI never invokes Codex or reads an OpenAI API key.