paper · benchmark creator

Assessment of Pharmaceutical Protein-Ligand Pose and Affinity Predictions in CASP16

University of California San Diego · University of Basel · University of California Davis · 2025-10-04

Relationship layer

Benchmark usage

This table records what the work did with each benchmark before attempting to normalize a run. Partial claims remain visible without being treated as comparable evaluations.

No BenchmarkUse relation is normalized for this legacy work yet. Existing EvaluationRuns remain available below.

Normalized evaluation runs

CASP Protein-Ligand Prediction3 runs

Open benchmark record →

Evaluation run

casp16-ligand-affinity-stage1

From Assessment of Pharmaceutical Protein-Ligand Pose and Affinity Predictions in CASP16

Scopesubset · n=122

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

Browsermethod-specific

Internetmethod-specific

Databasesmethod-specific

Code executionallowed

ContainerNot reported

External toolsmethod-specific

Token budgetNot applicable

Time / cost budgettarget-specific CASP submission windows before experimental poses were released

TemperatureNot applicable

SeedNot reported

RepeatsModel 1 affinity submission per group and case

Graderofficial affinity ranking analysis plus independent ligand assessor team · human review: yes

StatisticsN-weighted Kendall tau across the two protein systems after disclosure filtering

Contaminationexperimental poses and affinities withheld, with 18 previously disclosed affinities removed from analysis

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
N-weighted Kendall's tau	absolute	rank correlation	case-count-weighted mean of per-protein-system Kendall tau after disclosure filtering	Not reported

No numeric result rows are published yet; the verified protocol remains useful.

Evidence

section: Sections 2.2–2.5 and 3.3; Figure 8A (Original 140 cases, 18 disclosure exclusions, Stage-1 setting, Model 1, and N-weighted Kendall tau.) — supports /benchmark_version, /scope, /protocol, /metrics

Evaluation run

casp16-ligand-affinity-stage2

From Assessment of Pharmaceutical Protein-Ligand Pose and Affinity Predictions in CASP16

Scopesubset · n=103

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

Browsermethod-specific

Internetmethod-specific

Databasesmethod-specific

Code executionallowed

ContainerNot reported

External toolsmethod-specific

Token budgetNot applicable

Time / cost budget2024-08-07 through 2024-08-21 for the main Stage-2 challenge

TemperatureNot applicable

SeedNot reported

RepeatsModel 1 affinity submission per group and case

Graderofficial affinity ranking analysis plus independent ligand assessor team · human review: yes

Statisticsper-system Kendall tau and N-weighted cross-system summary after disclosure filtering

Contaminationexperimental co-crystal structures provided while experimental affinities remained withheld; disclosed affinities removed

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
N-weighted Kendall's tau	absolute	rank correlation	case-count-weighted mean of per-protein-system Kendall tau after disclosure filtering	Not reported

No numeric result rows are published yet; the verified protocol remains useful.

Evidence

section: Sections 2.2–2.5 and 3.3; Figures 8B-C and 9 (110 releases, 103 retained cases, co-crystal structure access, Stage-2 dates, and Kendall tau analysis.) — supports /benchmark_version, /scope, /protocol, /metrics

Evaluation run

casp16-ligand-pose-regular

From Assessment of Pharmaceutical Protein-Ligand Pose and Affinity Predictions in CASP16

Scopesubset · n=229

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

Browsermethod-specific

Internetmethod-specific

Databasesmethod-specific

Code executionallowed

ContainerNot reported

External toolsmethod-specific

Token budgetNot applicable

Time / cost budgettarget-specific CASP submission windows

TemperatureNot applicable

SeedNot reported

Repeatsup to five pose models per target

Graderofficial OpenStructure ligand and binding-pocket scoring plus independent ligand assessor team · human review: yes

Statisticsmean per-group pose accuracy with explicit handling of skipped targets; Model 1 is the primary ranking view

Contaminationexperimental complex poses withheld during Stage 1 prediction; structural template similarity analyzed post hoc

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
LDDT-PLI	absolute	proportion	mean across eligible pose targets with missing-prediction handling	CASP16 modified the CASP15 definition to penalize erroneous non-native contacts.
BiSyRMSD	absolute	angstrom	per pose then group summary	symmetry-corrected after binding-site superposition
LDDT-lp	absolute	proportion	per binding pocket then group summary	Not reported
BB-RMSD	absolute	angstrom	binding-site C-alpha RMSD then group summary	Not reported

No numeric result rows are published yet; the verified protocol remains useful.

Evidence

section: Abstract; Sections 2.1–2.5 and 3.1–3.2 (229 assessed pose cases, participant protocol, Model 1 handling, pose metrics, missing submissions, and separate baselines.) — supports /benchmark_version, /scope, /protocol, /metrics