Review a paper →
Submit a DOI/preprint plus a legal open or authorized source. The issue becomes a candidate until the owner explicitly starts a local Codex intake.
Add benchmark →
Creator source, access, counts, taxonomy, and runnable status.
Add evaluation →
Work, exact model, relationship, scope, metrics, tools, grader, and locators.
Correct metadata →
Show the current field, proposed value, and primary evidence.
Broken source →
Report moved, unavailable, or materially changed canonical sources.
What happens to a submitted paper
- DOI, preprint ID, canonical URL, title fingerprint, open candidate issues, and recent paper-intake PRs are checked for duplication.
- The source is MIME-checked, capped at 45 MiB / 150 pages, fingerprinted, and read only from an ignored local temporary directory.
- After the owner explicitly selects the paper, a fresh read-only Codex session separates evaluation, training/fine-tuning, validation, model selection, external-result summary, and pure background citations.
- A second fresh Codex session independently rechecks every claim and locator against the original source. Disagreement never becomes a normal field.
- Deterministic code—not either session—assigns IDs and creates Work, BenchmarkUse, Model, and sufficiently complete EvaluationRun records.
- A Ready PR runs normal CI and can merge only after
wang422003comments/approve-paper-intake <full-current-head-sha>.
Pull request checklist
- Describe what changed and cite a primary or official source.
- Map Scientific Tasks only from an official taxonomy, formal track, or locatable official artifact; preserve count unit and basis.
- Say whether the evaluation is full, subset, track, or unknown—and include realized
nwhen reported. - Record metrics, aggregation, tools, repeats, grader, and every remaining unreported field.
- Do not copy benchmark questions, restricted attachments, or third-party papers into this repository.
- Run
pnpm registry:validateandpnpm registry:build.
Review policy
All production changes use pull requests. Paper-intake PRs additionally require the paper-owner-gate: only an exact-SHA approval comment by wang422003 passes, and any later push invalidates it. Local Codex never auto-merges. Candidate issues, source files, evidence excerpts, session transcripts, and model drafts are excluded from public exports.