competition · audited-with-caveats · verified 2026-07-21

CAMEO

Weekly, automated, independent, blind evaluation of registered macromolecular structure-prediction servers on complete PDB entries whose experimental structures are withheld during prediction.

+1 more

Audited with caveats: 1 field(s) are marked provisional or conflicted. Warnings are shown next to affected values and these claims are excluded from unqualified comparisons.

Rolling-service audit: the current platform contains one category—complex structures modeling (3D)—and has no fixed all-time total. The bounded 2024 creator-paper snapshot contains 7,150 selected targets; every live-service claim is dated to 2026-07-21 and server comparisons use common target subsets.

Benchmark definition

What is counted

Version: current-complex-3d
Total: Not reported (selected complete PDB entries accumulated by an unbounded weekly rolling service)
Task formats: continuous blind complete-complex structure prediction; stoichiometry prediction; ligand pose prediction
Capabilities: Prediction
Modalities: Protein sequenceDNA or RNA sequence3D structure

Version history

Version	Status	Release / as-of	Total	Formal tracks
legacy-multicategory `cameo-legacy-multicategory`	superseded	2012-01-01	Not reported (rolling targets across now-discontinued single-chain 3D, quality-estimation, contact-prediction, and ligand-binding-site categories)	None registered
2024-complex-study `cameo-2024-complex-study`	superseded	2024-01-06	7150 (selected interesting complete PDB entries in the creator-paper study window)	None registered
current-complex-3d `cameo-current-complex-3d`	rolling	2026-07-21	Not reported (selected complete PDB entries accumulated by an unbounded weekly rolling service)	None registered

Scientific Task Atlas

Scientific task classification

partial for current-complex-3d · as of 2026-07-21. Rolling service; task mappings describe current public categories and never imply an all-time target total.

Scientific task	Coverage	Count	Mapping	Evidence
Protein complex structure prediction	explicitly-in-scope	Not reported Weekly complete-complex targets in the current CAMEO service.	official-taxonomy high confidence	`cameo-evidence-current-help` No fixed all-time count exists for the rolling service.
Protein model quality assessment	explicitly-in-scope	Not reported Weekly targets and submitted server models.	official-taxonomy high confidence	`cameo-evidence-2024-study` CAMEO evaluates prediction quality against newly released structures.
Protein-ligand pose prediction	explicitly-in-scope	Not reported Ligand-containing targets in the current rolling service.	official-taxonomy high confidence	`cameo-evidence-2024-study` Ligand pose is in scope; binding affinity is not asserted.

Scientific coverage notes

Domain	Coverage	Count	Interpretation
Protein-protein binding	explicitly-in-scope	Not reported	Medium and hard complete-complex targets include homomers, heteromers, and antibodies; the living service has no fixed all-time count.
Protein-ligand binding	explicitly-in-scope	Not reported	Ligand targets and ligand-containing medium/hard targets are in scope; the living service has no fixed all-time count.
Antibody-antigen	explicitly-in-scope	Not reported	Antibody complexes are analyzed as a protein-protein subclass; no current rolling count is asserted.

Evaluation registry

Works and run settings

A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.

Evaluation run

cameo-2024-antibody-three-server-common

From Beyond Single Chains: Benchmarking Macromolecular Complex Prediction Methods With the Continuous Automated Model EvaluatiOn (CAMEO)

Evaluated models / systems: AlphaFold 3 v3.0.1 (CAMEO baseline), MultiFOLD (CAMEO 2024 participant), SWISS-MODEL (CAMEO 2024 participant)

Scopesubset · n=83

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

Browsermethod-specific

Internetallowed before each weekly deadline

Databasesmethod-specific public structure and sequence data

Code executionallowed

ContainerNot reported

External toolsmethod-specific server pipeline

Token budgetNot applicable

Time / cost budgetapproximately 3.5 days per weekly target

TemperatureNot applicable

SeedNot reported

Repeatsup to five submitted models per target

Graderfully automated OpenStructure complex LDDT scoring · human review: no

Statisticsmedian Complex LDDT across 83 common-subset antibody targets

Contaminationexperimental antibody-complex structures withheld until PDB release

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Median Complex LDDT	absolute	proportion	median across 83 antibody common-subset targets	missing chains are penalized

Results

Model	Metric	Value	n
AlphaFold 3 v3.0.1 (CAMEO baseline)	Median Complex LDDT	0.83 proportion Median reported in creator-paper Section 2.4.2.	83
MultiFOLD (CAMEO 2024 participant)	Median Complex LDDT	0.76 proportion Median reported in creator-paper Section 2.4.2.	83

Evidence

section: Section 2.4.2 and Figure 2A (Reports 83 antibody common-subset targets, AF3 median LDDT 0.83, MultiFOLD median LDDT 0.76, and the model-1 comparison protocol.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics, /results

Evaluation run

cameo-2024-ligand-baseline-common

From Beyond Single Chains: Benchmarking Macromolecular Complex Prediction Methods With the Continuous Automated Model EvaluatiOn (CAMEO)

Evaluated models / systems: AlphaFold 3 v3.0.1 (CAMEO baseline), SWISS-MODEL + Schrödinger Glide, SWISS-MODEL + AutoDock Vina (AutoDock4 scoring), SWISS-MODEL + AutoDock Vina (vina scoring)

Scopesubset · n=2584

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

Browsermethod-specific

Internetallowed before each weekly deadline

Databasesmethod-specific public structure and sequence data

Code executionallowed

ContainerNot reported

External toolsAlphaFold 3 v3.0.1 or SWISS-MODEL followed by Glide or AutoDock Vina 1.2.5

Token budgetNot applicable

Time / cost budgetapproximately 3.5 days per weekly target

TemperatureNot applicable

SeedNot reported

Repeatsup to five models or ligand poses per target

Graderfully automated OpenStructure complex and ligand scoring · human review: no

Statisticscommon-subset aggregation across targets predicted by all four baselines

ContaminationPDB pre-release sequences and ligand identities available while experimental structures and poses remain withheld

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Ligand success rate	absolute	percent of ligand entities	percentage across common-subset ligand entities	A success is a symmetry-corrected BiSyRMSD below 2 Å after binding-site superposition.
LDDT-PLI	absolute	proportion	weighted aggregation across relevant ligand entities	Not reported
BiSyRMSD	absolute	angstrom	best-scored submitted pose per ligand entity in the creator analysis	2

No numeric result rows are published yet; the verified protocol remains useful.

Evidence

section: Sections 2.3, 2.4.1, 2.5.1-2.5.2, and 3.2; Figure 1 (Reports the four baselines, 2,584-target common subset, 6,152 entities, top-ranked model handling, ligand metrics, and exact pipeline versions.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics

Evaluation run

cameo-2024-ppi-three-server-common

From Beyond Single Chains: Benchmarking Macromolecular Complex Prediction Methods With the Continuous Automated Model EvaluatiOn (CAMEO)

Evaluated models / systems: AlphaFold 3 v3.0.1 (CAMEO baseline), MultiFOLD (CAMEO 2024 participant), SWISS-MODEL (CAMEO 2024 participant)

Scopesubset · n=392

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

Browsermethod-specific

Internetallowed before each weekly deadline

Databasesmethod-specific public structure and sequence data

Code executionallowed

ContainerNot reported

External toolsmethod-specific server pipeline

Token budgetNot applicable

Time / cost budgetapproximately 3.5 days per weekly target

TemperatureNot applicable

SeedNot reported

Repeatsup to five submitted models per target

Graderfully automated OpenStructure whole-complex and interface scoring · human review: no

Statisticsscore distributions on the three-server common subset

Contaminationexperimental complex structures withheld until the weekly PDB release

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Complex LDDT	absolute	proportion	per target distribution with penalties for missing chains	Not reported
Mapped complex LDDT	absolute	proportion	per target distribution over mapped chains	removes the missing-chain stoichiometry penalty
Complex iLDDT	absolute	proportion	per target inter-chain-contact distribution with stoichiometry penalties	Not reported
Mapped complex iLDDT	absolute	proportion	per target inter-chain contacts over mapped chains	removes the missing-chain stoichiometry penalty

No numeric result rows are published yet; the verified protocol remains useful.

Evidence

section: Sections 2.2, 2.4.2, and 2.5.1; Figure 2 (Reports the 392-target common subset, three systems, blind stoichiometry setting, model-1 analysis, and four metrics.) — supports /benchmark_version, /model_ids, /scope, /protocol, /metrics

Comparable result views

Median Complex LDDT

cameo-2024-antibody-three-server-common · cameo-2024-antibody-three-server-common

CSV ↓

Accessible data table

Model	Value	Comparability group
AlphaFold 3 v3.0.1 (CAMEO baseline)	0.83	cameo-2024-antibody-three-server-common
MultiFOLD (CAMEO 2024 participant)	0.76	cameo-2024-antibody-three-server-common

Evidence and change history

Source locators remain visible; expand an item to inspect the exact Registry fields it supports.

Continuous Automated Model EvaluatiOn (CAMEO) Complementing the Critical Assessment of Structure Prediction in CASP12 · section: Abstract and Introduction (Documents the creator-operated continuous platform and its weekly blind PDB pre-release workflow since 2012.) · Supports 5 fields

Open source →

/name
/aliases
/kind
/organizations
/release_date

cameo-legacy-help-resource · section: Archive notice and General Workflow (Identifies the historical categories and the April 2025 end of single-chain 3D.) · Supports 4 fields

Open source →

/versions/0/as_of
/versions/0/task_counts/total
/versions/0/task_counts/basis
/versions/0/task_counts/subsets

Beyond Single Chains: Benchmarking Macromolecular Complex Prediction Methods With the Continuous Automated Model EvaluatiOn (CAMEO) · section: Sections 2.1-2.5 and 3; Figures 1-4 (Reports the 2024 window, 14,078 PDB releases, 7,150 selected targets, category counts, common subsets, models, metrics, and public downloads.) · Supports 15 fields

Open source →

/summary
/domains
/capabilities
/modalities
/task_formats
/coverage_notes
/versions/1/release_date
/versions/1/as_of
/versions/1/task_counts/total
/versions/1/task_counts/basis
/versions/1/task_counts/subsets
/resources
/implementations
/scientific_task_classification/entries/1
/scientific_task_classification/entries/2

cameo-current-help-resource · section: General Workflow; Target Submission; Prediction Format; Evaluation; Scores aggregation (Defines the current complex-only rolling category, target unit, four-day window, up-to-five models, OpenStructure scoring, common-subset aggregation, and access.) · Supports 13 fields

Open source →

/latest_version
/task_counts/total
/task_counts/basis
/task_counts/subsets
/access/tasks
/access/artifacts
/access/grader
/access/biosafety_notes
/versions/2/as_of
/versions/2/task_counts/total
/versions/2/task_counts/basis
/versions/2/task_counts/subsets
/scientific_task_classification/entries/0

cameo-terms-resource · section: Description; Terms; Copyright (Public versus development-server access and CC BY-SA 4.0 redistribution terms.) · Supports 2 fields

Open source →

/access/level
/access/license

Unresolved field claims

/release_dateProvisional · medium — Creator sources establish that CAMEO has operated since 2012 but do not report an exact first service day; 2012-01-01 is a year-normalized date.
Evidence: cameo-evidence-origin

View source-level modification history on GitHub →

Access & reproducibility

Access: Partially open
License: CAMEO-provided data and submitted redistributable predictions are CC BY-SA 4.0; AlphaFold 3 outputs retain separate terms.
Grader: Fully automated OpenStructure scoring against released PDB biounits; no single metric or over…
Implementations: 1 recorded · official

Access details

Tasks: Public and development servers receive selected PDB pre-release entries; participation requires server registration and development-server names remain disguised.
Artifacts: Public-server targets, model-1 results, reference structures, predictions, scores, and time-window downloads are public after release; models 2-5 and advanced scores are in downloads.
Grader: Fully automated OpenStructure scoring against released PDB biounits; no single metric or overall ranking is defined.
Biosafety: The registry mirrors no pre-release sequences, structures, ligands, predictions, or development-server results.

Resources & implementations

Primary resources

cameo-current-site-resource · CC BY-SA 4.0 for CAMEO-provided data
website · snapshot 2026-07-21 · checked 2026-07-21
cameo-current-help-resource · CC BY-SA 4.0 for CAMEO-provided data
documentation · snapshot 2026-07-21 · checked 2026-07-21
cameo-terms-resource · CC BY-SA 4.0
documentation · snapshot 2026-07-21 · checked 2026-07-21
cameo-complex-paper-resource · CC BY 4.0
paper · version Proteins 94(1):403-413 · checked 2026-07-21
cameo-foundation-paper-resource
paper · version Proteins 86(S1):387-398 · checked 2026-07-21
cameo-legacy-help-resource · CC BY-SA 4.0 for CAMEO-provided data
documentation · snapshot legacy-single-chain-ended-2025-04 · checked 2026-07-21

Implementations

CAMEO hosted evaluation service using OpenStructure
official