Evaluation run
proteingym-v10-dms-substitutions-zero-shot
From ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
Evaluated models / systems: CARP (38M), CARP (600K), CARP (640M), CARP (76M), DeepSequence (ensemble), DeepSequence (single), ESM-1b, ESM-1v (ensemble), ESM-1v (single), ESM-IF1, ESM2 (150M), ESM2 (15B), ESM2 (35M), ESM2 (3B), ESM2 (650M), ESM2 (8M), EVE (ensemble), EVE (single), EVmutation, GEMME, MIF, MIF-ST, MSA Transformer (ensemble), MSA Transformer (single), ProGen2 Base, ProGen2 L, ProGen2 M, ProGen2 S, ProGen2 XL, ProteinMPNN, ProtGPT2, RITA L, RITA M, RITA S, RITA XL, Site-Independent, TranceptEVE L, TranceptEVE M, TranceptEVE S, Tranception L, Tranception L no retrieval, Tranception M, Tranception M no retrieval, Tranception S, Tranception S no retrieval, UniRep, UniRep evotuned, VESPA, VESPAl, WaveNet
Scopefull · n=217
Shots0
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific inputs
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic official scoring pipeline · human review: no
StatisticsNon-parametric bootstrap standard error of each model-to-best Spearman difference over 10,000 bootstrap samples from proteins
ContaminationNot reported
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman | absolute | correlation | assay-level Spearman, averaged within five functional groups, then equal-weight mean across groups | Not reported |
| AUC | absolute | area under ROC curve | assay-level AUC, averaged within five functional groups, then equal-weight mean across groups | Not reported |
| MCC | absolute | correlation | assay-level MCC, averaged within five functional groups, then equal-weight mean across groups | Not reported |
| NDCG@10% | absolute | normalized discounted cumulative gain | assay-level NDCG@10%, averaged within five functional groups, then equal-weight mean across groups | Not reported |
| Top 10% recall | absolute | proportion | assay-level top-10% recall, averaged within five functional groups, then equal-weight mean across groups | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| TranceptEVE L | Spearman | 0.456 correlation Table A5 rank 1*; bootstrap SE of difference from the best model: 0.000. | Not reported |
| TranceptEVE M | Spearman | 0.455 correlation Table A5 rank 1*; bootstrap SE of difference from the best model: 0.004. | Not reported |
| GEMME | Spearman | 0.455 correlation Table A5 rank 1*; bootstrap SE of difference from the best model: 0.007. | Not reported |
| TranceptEVE S | Spearman | 0.452 correlation Table A5 rank 4; bootstrap SE of difference from the best model: 0.004. | Not reported |
| EVE (ensemble) | Spearman | 0.439 correlation Table A5 rank 5; bootstrap SE of difference from the best model: 0.006. | Not reported |
| VESPA | Spearman | 0.436 correlation Table A5 rank 6; bootstrap SE of difference from the best model: 0.006. | Not reported |
| Tranception L | Spearman | 0.434 correlation Table A5 rank 7*; bootstrap SE of difference from the best model: 0.004. | Not reported |
| MSA Transformer (ensemble) | Spearman | 0.434 correlation Table A5 rank 7*; bootstrap SE of difference from the best model: 0.009. | Not reported |
| EVE (single) | Spearman | 0.433 correlation Table A5 rank 9; bootstrap SE of difference from the best model: 0.005. | Not reported |
| Tranception M | Spearman | 0.427 correlation Table A5 rank 10; bootstrap SE of difference from the best model: 0.005. | Not reported |
| ESM-IF1 | Spearman | 0.422 correlation Table A5 rank 11; bootstrap SE of difference from the best model: 0.011. | Not reported |
| MSA Transformer (single) | Spearman | 0.421 correlation Table A5 rank 12; bootstrap SE of difference from the best model: 0.009. | Not reported |
| DeepSequence (ensemble) | Spearman | 0.419 correlation Table A5 rank 13; bootstrap SE of difference from the best model: 0.008. | Not reported |
| Tranception S | Spearman | 0.418 correlation Table A5 rank 14; bootstrap SE of difference from the best model: 0.006. | Not reported |
| ESM2 (650M) | Spearman | 0.414 correlation Table A5 rank 15; bootstrap SE of difference from the best model: 0.012. | Not reported |
| DeepSequence (single) | Spearman | 0.407 correlation Table A5 rank 16*; bootstrap SE of difference from the best model: 0.008. | Not reported |
| ESM-1v (ensemble) | Spearman | 0.407 correlation Table A5 rank 16*; bootstrap SE of difference from the best model: 0.012. | Not reported |
| ESM2 (3B) | Spearman | 0.406 correlation Table A5 rank 18; bootstrap SE of difference from the best model: 0.011. | Not reported |
| MIF-ST | Spearman | 0.401 correlation Table A5 rank 19*; bootstrap SE of difference from the best model: 0.010. | Not reported |
| ESM2 (15B) | Spearman | 0.401 correlation Table A5 rank 19*; bootstrap SE of difference from the best model: 0.010. | Not reported |
| EVmutation | Spearman | 0.395 correlation Table A5 rank 21; bootstrap SE of difference from the best model: 0.006. | Not reported |
| ESM-1b | Spearman | 0.394 correlation Table A5 rank 22*; bootstrap SE of difference from the best model: 0.010. | Not reported |
| VESPAl | Spearman | 0.394 correlation Table A5 rank 22*; bootstrap SE of difference from the best model: 0.007. | Not reported |
| ProGen2 XL | Spearman | 0.391 correlation Table A5 rank 24; bootstrap SE of difference from the best model: 0.008. | Not reported |
| ESM2 (150M) | Spearman | 0.387 correlation Table A5 rank 25; bootstrap SE of difference from the best model: 0.013. | Not reported |
| MIF | Spearman | 0.382 correlation Table A5 rank 26; bootstrap SE of difference from the best model: 0.011. | Not reported |
| ProGen2 L | Spearman | 0.38 correlation Table A5 rank 27; bootstrap SE of difference from the best model: 0.008. | Not reported |
| ProGen2 M | Spearman | 0.379 correlation Table A5 rank 28; bootstrap SE of difference from the best model: 0.008. | Not reported |
| ProGen2 Base | Spearman | 0.378 correlation Table A5 rank 29; bootstrap SE of difference from the best model: 0.009. | Not reported |
| Tranception L no retrieval | Spearman | 0.374 correlation Table A5 rank 30*; bootstrap SE of difference from the best model: 0.008. | Not reported |
| ESM-1v (single) | Spearman | 0.374 correlation Table A5 rank 30*; bootstrap SE of difference from the best model: 0.013. | Not reported |
| WaveNet | Spearman | 0.373 correlation Table A5 rank 32; bootstrap SE of difference from the best model: 0.012. | Not reported |
| RITA XL | Spearman | 0.372 correlation Table A5 rank 33; bootstrap SE of difference from the best model: 0.009. | Not reported |
| CARP (640M) | Spearman | 0.368 correlation Table A5 rank 34; bootstrap SE of difference from the best model: 0.011. | Not reported |
| RITA L | Spearman | 0.365 correlation Table A5 rank 35; bootstrap SE of difference from the best model: 0.009. | Not reported |
| Site-Independent | Spearman | 0.359 correlation Table A5 rank 36; bootstrap SE of difference from the best model: 0.010. | Not reported |
| RITA M | Spearman | 0.35 correlation Table A5 rank 37; bootstrap SE of difference from the best model: 0.010. | Not reported |
| Tranception M no retrieval | Spearman | 0.348 correlation Table A5 rank 38; bootstrap SE of difference from the best model: 0.009. | Not reported |
| UniRep evotuned | Spearman | 0.347 correlation Table A5 rank 39; bootstrap SE of difference from the best model: 0.009. | Not reported |
| ProGen2 S | Spearman | 0.336 correlation Table A5 rank 40; bootstrap SE of difference from the best model: 0.012. | Not reported |
| CARP (76M) | Spearman | 0.328 correlation Table A5 rank 41; bootstrap SE of difference from the best model: 0.012. | Not reported |
| ESM2 (35M) | Spearman | 0.321 correlation Table A5 rank 42; bootstrap SE of difference from the best model: 0.015. | Not reported |
| RITA S | Spearman | 0.304 correlation Table A5 rank 43; bootstrap SE of difference from the best model: 0.011. | Not reported |
| Tranception S no retrieval | Spearman | 0.303 correlation Table A5 rank 44; bootstrap SE of difference from the best model: 0.012. | Not reported |
| CARP (38M) | Spearman | 0.279 correlation Table A5 rank 45; bootstrap SE of difference from the best model: 0.014. | Not reported |
| ProteinMPNN | Spearman | 0.258 correlation Table A5 rank 46; bootstrap SE of difference from the best model: 0.011. | Not reported |
| ESM2 (8M) | Spearman | 0.226 correlation Table A5 rank 47; bootstrap SE of difference from the best model: 0.015. | Not reported |
| UniRep | Spearman | 0.19 correlation Table A5 rank 48; bootstrap SE of difference from the best model: 0.016. | Not reported |
| ProtGPT2 | Spearman | 0.188 correlation Table A5 rank 49; bootstrap SE of difference from the best model: 0.011. | Not reported |
| CARP (600K) | Spearman | 0.106 correlation Table A5 rank 50; bootstrap SE of difference from the best model: 0.016. | Not reported |
Evidence
- section: pp. 5–8 and 29, Table 1, Sections 3.3–4.1, and Table A1 (Defines v1.0, 217 substitution assays, zero-shot label access, baselines, and official scoring.) — supports /scope, /protocol
- table: pp. 3, 6, 9, 35, and 38; Figure 1, Section 4.1, Table 2, Appendix A.5.1, and Table A5 (Defines Spearman, AUC, MCC, NDCG@10%, top-10% recall, corrected-average aggregation, and 10,000-sample bootstrap comparisons.) — supports /metrics, /protocol/statistical
- table: p. 38, Table A5 (All 50 zero-shot DMS substitution Spearman values and bootstrap standard errors of model-to-best differences.) — supports /results