FLIP AAV
Seven supervised splits over sampled and machine-designed AAV2 VP-1 capsid variants, measuring generalization across mutation depth, fitness, and sampled-versus-designed pools.
7 evaluation run(s)
suite · audited · verified 2026-07-21
FLIP
A supervised protein sequence-to-fitness benchmark that turns three experimental landscapes into 15 biologically motivated dataset splits for testing generalization in protein engineering.
Task-unit audit: original FLIP contains 15 dataset×split tasks across AAV (7), GB1 (5), and Meltome (3). Thirteen splits are active for performance comparison; the two random Sampled splits are retained as discourse-only and never share a comparability group with active splits.
Benchmark definition
What is counted
- Version
- original-2021
- Total
- 15 (dataset-and-split benchmark tasks)
- Task formats
- supervised sequence-to-fitness regression
- Capabilities
- Modalities
Version history
| Version | Status | Release / as-of | Total | Formal tracks |
|---|---|---|---|---|
original-2021flip-original-2021 | current | 2021-10-11 | 15 (dataset-and-split benchmark tasks) | flip-aav, flip-gb1, flip-meltome |
Tracks and subsets
| ID | Count | Basis | Partition? | Notes |
|---|---|---|---|---|
AAV landscape splitsflip-aav-tasks | 7 | dataset-and-split tasks | Exclusive & exhaustive | Six active comparison splits plus one sampled split used mainly for discourse. |
GB1 landscape splitsflip-gb1-tasks | 5 | dataset-and-split tasks | Exclusive & exhaustive | Four active comparison splits plus one sampled split used mainly for discourse. |
Meltome thermostability splitsflip-meltome-tasks | 3 | dataset-and-split tasks | Exclusive & exhaustive | Mixed, Human, and Human-cell are all active. |
Active performance-comparison splitsflip-active-comparison-tasks | 13 | dataset-and-split tasks | No | Current official repository semaphore marks all original splits active except the two sampled splits. |
Sampled discourse splitsflip-discourse-sampled-tasks | 2 | dataset-and-split tasks | No | AAV Sampled and GB1 Sampled are orange: the repository warns against performance comparisons because random sampling can overestimate performance. |
Registered child tracks
FLIP GB1
Five supervised splits over a downsampled, highly epistatic four-site GB1 immunoglobulin-binding landscape, designed to test mutation-depth and low-to-high-fitness generalization.
5 evaluation run(s)
FLIP Meltome Thermostability
Three supervised sequence-to-melting-temperature splits spanning all species, human proteins, and a single human cell line, with sequence-cluster-aware train/test separation.
3 evaluation run(s)
Scientific Task Atlas
Scientific task classification
complete for original-2021. FLIP evaluates sequence-to-fitness prediction; it is not a sequence-generation benchmark.
| Scientific task | Coverage | Count | Mapping | Evidence |
|---|---|---|---|---|
| Protein fitness prediction | explicitly-in-scope | 15 tasks Dataset-and-split benchmark tasks. | official-taxonomy high confidence | flip-evidence-paper-definitionAll formal tasks are supervised fitness or phenotype prediction. |
| Protein mutation-effect prediction | explicitly-in-scope | 15 tasks Dataset-and-split benchmark tasks. | official-taxonomy high confidence | flip-evidence-paper-definitionThe same tasks evaluate generalization across mutated sequence landscapes; claims overlap and are never summed. |
Scientific coverage notes
| Domain | Coverage | Count | Interpretation |
|---|---|---|---|
| Protein-protein binding | explicitly-in-scope | 5 | All five GB1 dataset-split tasks predict the fitness of an immunoglobulin-binding protein domain; this is task count, not a distinct set of five assays. |
| Protein design | explicitly-in-scope | Not reported | All tasks are motivated by protein engineering, but FLIP evaluates prediction/regression rather than sequence generation or optimization and does not publish a separate design-task count. |
Evaluation registry
Works and run settings
A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.
Evaluation run
flip-aav-des-mut
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=82583
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationPool shift: model-designed variants are training data and sampled variants are held out.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.59 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.7 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.44 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.71 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| ESM-untrained (mean) | Spearman correlation | 0.34 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| ESM-untrained (mut mean) | Spearman correlation | 0.64 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| Ridge regression (FLIP) | Spearman correlation | 0.53 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| Convolutional network (FLIP) | Spearman correlation | 0.75 correlation Held-out test-set Spearman from the creator paper. | 82583 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | -0.07 correlation Held-out test-set Spearman from the creator paper. | 82583 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 5 (Des-Mut column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-aav-low-vs-high
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=35037
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationFitness extrapolation above wild-type fitness.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.39 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.18 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.33 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.34 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.2 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.31 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-untrained (per AA) | Spearman correlation | 0.08 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-untrained (mean) | Spearman correlation | 0.22 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| ESM-untrained (mut mean) | Spearman correlation | 0.24 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| Ridge regression (FLIP) | Spearman correlation | 0.12 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| Convolutional network (FLIP) | Spearman correlation | 0.34 correlation Held-out test-set Spearman from the creator paper. | 35037 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | 0.25 correlation Held-out test-set Spearman from the creator paper. | 35037 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 5 (low-vs-high column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-aav-mut-des
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=201426
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationPool shift: sampled variants are training data and model-designed variants are held out.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.76 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.63 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.7 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.79 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.55 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.7 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-untrained (per AA) | Spearman correlation | 0.56 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-untrained (mean) | Spearman correlation | 0.27 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| ESM-untrained (mut mean) | Spearman correlation | 0.62 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| Ridge regression (FLIP) | Spearman correlation | 0.64 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| Convolutional network (FLIP) | Spearman correlation | 0.71 correlation Held-out test-set Spearman from the creator paper. | 201426 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | 0.6 correlation Held-out test-set Spearman from the creator paper. | 201426 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 5 (Mut-Des column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-aav-one-vs-rest
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=81413
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationMutation-depth extrapolation; the held-out set excludes wild type and single-mutant training examples.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.03 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.04 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.31 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.1 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.18 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.44 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-untrained (per AA) | Spearman correlation | 0.18 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-untrained (mean) | Spearman correlation | 0.01 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| ESM-untrained (mut mean) | Spearman correlation | 0.26 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| Ridge regression (FLIP) | Spearman correlation | 0.22 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| Convolutional network (FLIP) | Spearman correlation | 0.48 correlation Held-out test-set Spearman from the creator paper. | 81413 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | -0.11 correlation Held-out test-set Spearman from the creator paper. | 81413 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 5 (1-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-aav-sampled
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (per AA), ESM-1b (per AA; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)
Scopesubset · n=16517
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationRandom split can place closely related variants across train/test and is explicitly described as optimistic.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.9 correlation Held-out test-set Spearman from the creator paper. | 16517 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.92 correlation Held-out test-set Spearman from the creator paper. | 16517 |
| ESM-untrained (per AA) | Spearman correlation | 0.78 correlation Held-out test-set Spearman from the creator paper. | 16517 |
| Ridge regression (FLIP) | Spearman correlation | 0.83 correlation Held-out test-set Spearman from the creator paper. | 16517 |
| Convolutional network (FLIP) | Spearman correlation | 0.92 correlation Held-out test-set Spearman from the creator paper. | 16517 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 10, Table 7 (AAV column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-aav-seven-vs-rest
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=12581
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationMutation-depth extrapolation above seven changes.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.65 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.46 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.61 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.7 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.45 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.64 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-untrained (per AA) | Spearman correlation | 0.42 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-untrained (mean) | Spearman correlation | 0.22 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| ESM-untrained (mut mean) | Spearman correlation | 0.56 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| Ridge regression (FLIP) | Spearman correlation | 0.65 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| Convolutional network (FLIP) | Spearman correlation | 0.74 correlation Held-out test-set Spearman from the creator paper. | 12581 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | 0.53 correlation Held-out test-set Spearman from the creator paper. | 12581 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 5 (7-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-aav-two-vs-rest
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=50776
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationMutation-depth extrapolation above two changes.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.65 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.26 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.65 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.7 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.16 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.64 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-untrained (per AA) | Spearman correlation | 0.22 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-untrained (mean) | Spearman correlation | 0.14 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| ESM-untrained (mut mean) | Spearman correlation | 0.16 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| Ridge regression (FLIP) | Spearman correlation | 0.03 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| Convolutional network (FLIP) | Spearman correlation | 0.74 correlation Held-out test-set Spearman from the creator paper. | 50776 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | 0.57 correlation Held-out test-set Spearman from the creator paper. | 50776 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 5 (2-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-gb1-low-vs-high
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: BLOSUM62 baseline (FLIP), Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=3644
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationFitness extrapolation above wild-type fitness.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.59 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.13 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.45 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.51 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.1 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.49 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-untrained (per AA) | Spearman correlation | 0.23 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-untrained (mean) | Spearman correlation | 0.1 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| ESM-untrained (mut mean) | Spearman correlation | 0.13 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| Ridge regression (FLIP) | Spearman correlation | 0.34 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| Convolutional network (FLIP) | Spearman correlation | 0.51 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | -0.1 correlation Held-out test-set Spearman from the creator paper. | 3644 |
| BLOSUM62 baseline (FLIP) | Spearman correlation | -0.13 correlation Held-out test-set Spearman from the creator paper. | 3644 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 8, Table 4 (low-vs-high column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-gb1-one-vs-rest
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: BLOSUM62 baseline (FLIP), Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=8704
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationMutation-depth extrapolation from wild type and single mutants.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.28 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.32 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | -0.08 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.28 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.32 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.19 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-untrained (per AA) | Spearman correlation | 0.06 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-untrained (mean) | Spearman correlation | 0.05 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| ESM-untrained (mut mean) | Spearman correlation | 0.21 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| Ridge regression (FLIP) | Spearman correlation | 0.28 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| Convolutional network (FLIP) | Spearman correlation | 0.17 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | 0.17 correlation Held-out test-set Spearman from the creator paper. | 8704 |
| BLOSUM62 baseline (FLIP) | Spearman correlation | 0.15 correlation Held-out test-set Spearman from the creator paper. | 8704 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 8, Table 4 (1-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-gb1-sampled
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (per AA), ESM-1b (per AA; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)
Scopesubset · n=1772
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationRandom split can place closely related variants across train/test and is explicitly described as optimistic.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.92 correlation Held-out test-set Spearman from the creator paper. | 1772 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.92 correlation Held-out test-set Spearman from the creator paper. | 1772 |
| ESM-untrained (per AA) | Spearman correlation | 0.79 correlation Held-out test-set Spearman from the creator paper. | 1772 |
| Ridge regression (FLIP) | Spearman correlation | 0.82 correlation Held-out test-set Spearman from the creator paper. | 1772 |
| Convolutional network (FLIP) | Spearman correlation | 0.91 correlation Held-out test-set Spearman from the creator paper. | 1772 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 10, Table 7 (GB1 column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-gb1-three-vs-rest
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: BLOSUM62 baseline (FLIP), Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=5765
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationMutation-depth extrapolation to four simultaneous mutations.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.79 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.54 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.49 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.82 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.77 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.8 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-untrained (per AA) | Spearman correlation | 0.48 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-untrained (mean) | Spearman correlation | 0.46 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| ESM-untrained (mut mean) | Spearman correlation | 0.57 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| Ridge regression (FLIP) | Spearman correlation | 0.76 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| Convolutional network (FLIP) | Spearman correlation | 0.83 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | -0.04 correlation Held-out test-set Spearman from the creator paper. | 5765 |
| BLOSUM62 baseline (FLIP) | Spearman correlation | 0.01 correlation Held-out test-set Spearman from the creator paper. | 5765 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 8, Table 4 (3-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-gb1-two-vs-rest
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: BLOSUM62 baseline (FLIP), Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)
Scopesubset · n=8306
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationMutation-depth extrapolation above two mutations.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.55 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.36 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-1b (mut mean; FLIP head) | Spearman correlation | 0.19 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.28 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.32 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-1v (mut mean; FLIP head) | Spearman correlation | 0.19 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-untrained (per AA) | Spearman correlation | 0.06 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-untrained (mean) | Spearman correlation | 0.05 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| ESM-untrained (mut mean) | Spearman correlation | 0.21 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| Ridge regression (FLIP) | Spearman correlation | 0.59 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| Convolutional network (FLIP) | Spearman correlation | 0.32 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| Levenshtein distance baseline (FLIP) | Spearman correlation | 0.16 correlation Held-out test-set Spearman from the creator paper. | 8306 |
| BLOSUM62 baseline (FLIP) | Spearman correlation | 0.14 correlation Held-out test-set Spearman from the creator paper. | 8306 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 8, Table 4 (2-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-meltome-human
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)
Scopesubset · n=1945
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationHuman-only sequence-cluster holdout at 20% identity.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.71 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.7 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.77 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.75 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| ESM-untrained (per AA) | Spearman correlation | 0.44 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| ESM-untrained (mean) | Spearman correlation | 0.48 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| Ridge regression (FLIP) | Spearman correlation | 0.15 correlation Held-out test-set Spearman from the creator paper. | 1945 |
| Convolutional network (FLIP) | Spearman correlation | 0.5 correlation Held-out test-set Spearman from the creator paper. | 1945 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 6 (Human column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-meltome-human-cell
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)
Scopesubset · n=1366
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationSingle-cell-line sequence-cluster holdout at 20% identity.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.76 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.75 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.78 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.74 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| ESM-untrained (per AA) | Spearman correlation | 0.46 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| ESM-untrained (mean) | Spearman correlation | 0.49 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| Ridge regression (FLIP) | Spearman correlation | 0.24 correlation Held-out test-set Spearman from the creator paper. | 1366 |
| Convolutional network (FLIP) | Spearman correlation | 0.49 correlation Held-out test-set Spearman from the creator paper. | 1366 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 6 (Human-Cell column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Evaluation run
flip-meltome-mixed
From FLIP: Benchmark tasks in fitness landscape inference for proteins
Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)
Scopesubset · n=3134
ShotsNot applicable
TurnsNot applicable
System prompt publicNot applicable
Reasoning / effortNot applicable
BrowserNot applicable
InternetNot applicable
DatabasesNot applicable
Code executionNot applicable
ContainerNot reported
External toolsmodel-specific frozen embeddings or one-hot sequence encoding
Token budgetNot applicable
Time / cost budgetNot reported
TemperatureNot applicable
SeedNot reported
RepeatsNot reported
Graderdeterministic regression scorer · human review: no
StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.
ContaminationSequence-cluster holdout at 20% identity; test uses held-out cluster representatives.
Metrics, results, and full protocol
Metrics
| Metric | Kind / baseline | Unit | Aggregation | Threshold / tolerance |
|---|---|---|---|---|
| Spearman correlation | absolute | correlation | computed across all examples in this held-out test split | Not reported |
| Mean squared error | absolute | squared fitness units | mean across all examples in this held-out test split | Not reported |
Results
| Model | Metric | Value | n |
|---|---|---|---|
| ESM-1b (per AA; FLIP head) | Spearman correlation | 0.68 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| ESM-1b (mean; FLIP head) | Spearman correlation | 0.68 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| ESM-1v (per AA; FLIP head) | Spearman correlation | 0.65 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| ESM-1v (mean; FLIP head) | Spearman correlation | 0.67 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| ESM-untrained (per AA) | Spearman correlation | 0.44 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| ESM-untrained (mean) | Spearman correlation | 0.36 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| Ridge regression (FLIP) | Spearman correlation | 0.17 correlation Held-out test-set Spearman from the creator paper. | 3134 |
| Convolutional network (FLIP) | Spearman correlation | 0.34 correlation Held-out test-set Spearman from the creator paper. | 3134 |
Evidence
- table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
- table: p. 9, Table 6 (Mixed column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results
Comparable result views
Spearman correlation
flip-aav-des-mut · flip-aav-des-mut-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (mean; FLIP head) | 0.59 | flip-aav-des-mut-spearman |
| ESM-1b (mut mean; FLIP head) | 0.7 | flip-aav-des-mut-spearman |
| ESM-1v (mean; FLIP head) | 0.44 | flip-aav-des-mut-spearman |
| ESM-1v (mut mean; FLIP head) | 0.71 | flip-aav-des-mut-spearman |
| ESM-untrained (mean) | 0.34 | flip-aav-des-mut-spearman |
| ESM-untrained (mut mean) | 0.64 | flip-aav-des-mut-spearman |
| Ridge regression (FLIP) | 0.53 | flip-aav-des-mut-spearman |
| Convolutional network (FLIP) | 0.75 | flip-aav-des-mut-spearman |
| Levenshtein distance baseline (FLIP) | -0.07 | flip-aav-des-mut-spearman |
Spearman correlation
flip-aav-low-vs-high · flip-aav-low-vs-high-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.39 | flip-aav-low-vs-high-spearman |
| ESM-1b (mean; FLIP head) | 0.18 | flip-aav-low-vs-high-spearman |
| ESM-1b (mut mean; FLIP head) | 0.33 | flip-aav-low-vs-high-spearman |
| ESM-1v (per AA; FLIP head) | 0.34 | flip-aav-low-vs-high-spearman |
| ESM-1v (mean; FLIP head) | 0.2 | flip-aav-low-vs-high-spearman |
| ESM-1v (mut mean; FLIP head) | 0.31 | flip-aav-low-vs-high-spearman |
| ESM-untrained (per AA) | 0.08 | flip-aav-low-vs-high-spearman |
| ESM-untrained (mean) | 0.22 | flip-aav-low-vs-high-spearman |
| ESM-untrained (mut mean) | 0.24 | flip-aav-low-vs-high-spearman |
| Ridge regression (FLIP) | 0.12 | flip-aav-low-vs-high-spearman |
| Convolutional network (FLIP) | 0.34 | flip-aav-low-vs-high-spearman |
| Levenshtein distance baseline (FLIP) | 0.25 | flip-aav-low-vs-high-spearman |
Spearman correlation
flip-aav-mut-des · flip-aav-mut-des-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.76 | flip-aav-mut-des-spearman |
| ESM-1b (mean; FLIP head) | 0.63 | flip-aav-mut-des-spearman |
| ESM-1b (mut mean; FLIP head) | 0.7 | flip-aav-mut-des-spearman |
| ESM-1v (per AA; FLIP head) | 0.79 | flip-aav-mut-des-spearman |
| ESM-1v (mean; FLIP head) | 0.55 | flip-aav-mut-des-spearman |
| ESM-1v (mut mean; FLIP head) | 0.7 | flip-aav-mut-des-spearman |
| ESM-untrained (per AA) | 0.56 | flip-aav-mut-des-spearman |
| ESM-untrained (mean) | 0.27 | flip-aav-mut-des-spearman |
| ESM-untrained (mut mean) | 0.62 | flip-aav-mut-des-spearman |
| Ridge regression (FLIP) | 0.64 | flip-aav-mut-des-spearman |
| Convolutional network (FLIP) | 0.71 | flip-aav-mut-des-spearman |
| Levenshtein distance baseline (FLIP) | 0.6 | flip-aav-mut-des-spearman |
Spearman correlation
flip-aav-one-vs-rest · flip-aav-one-vs-rest-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.03 | flip-aav-one-vs-rest-spearman |
| ESM-1b (mean; FLIP head) | 0.04 | flip-aav-one-vs-rest-spearman |
| ESM-1b (mut mean; FLIP head) | 0.31 | flip-aav-one-vs-rest-spearman |
| ESM-1v (per AA; FLIP head) | 0.1 | flip-aav-one-vs-rest-spearman |
| ESM-1v (mean; FLIP head) | 0.18 | flip-aav-one-vs-rest-spearman |
| ESM-1v (mut mean; FLIP head) | 0.44 | flip-aav-one-vs-rest-spearman |
| ESM-untrained (per AA) | 0.18 | flip-aav-one-vs-rest-spearman |
| ESM-untrained (mean) | 0.01 | flip-aav-one-vs-rest-spearman |
| ESM-untrained (mut mean) | 0.26 | flip-aav-one-vs-rest-spearman |
| Ridge regression (FLIP) | 0.22 | flip-aav-one-vs-rest-spearman |
| Convolutional network (FLIP) | 0.48 | flip-aav-one-vs-rest-spearman |
| Levenshtein distance baseline (FLIP) | -0.11 | flip-aav-one-vs-rest-spearman |
Spearman correlation
flip-aav-sampled · flip-aav-sampled-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.9 | flip-aav-sampled-spearman |
| ESM-1v (per AA; FLIP head) | 0.92 | flip-aav-sampled-spearman |
| ESM-untrained (per AA) | 0.78 | flip-aav-sampled-spearman |
| Ridge regression (FLIP) | 0.83 | flip-aav-sampled-spearman |
| Convolutional network (FLIP) | 0.92 | flip-aav-sampled-spearman |
Spearman correlation
flip-aav-seven-vs-rest · flip-aav-seven-vs-rest-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.65 | flip-aav-seven-vs-rest-spearman |
| ESM-1b (mean; FLIP head) | 0.46 | flip-aav-seven-vs-rest-spearman |
| ESM-1b (mut mean; FLIP head) | 0.61 | flip-aav-seven-vs-rest-spearman |
| ESM-1v (per AA; FLIP head) | 0.7 | flip-aav-seven-vs-rest-spearman |
| ESM-1v (mean; FLIP head) | 0.45 | flip-aav-seven-vs-rest-spearman |
| ESM-1v (mut mean; FLIP head) | 0.64 | flip-aav-seven-vs-rest-spearman |
| ESM-untrained (per AA) | 0.42 | flip-aav-seven-vs-rest-spearman |
| ESM-untrained (mean) | 0.22 | flip-aav-seven-vs-rest-spearman |
| ESM-untrained (mut mean) | 0.56 | flip-aav-seven-vs-rest-spearman |
| Ridge regression (FLIP) | 0.65 | flip-aav-seven-vs-rest-spearman |
| Convolutional network (FLIP) | 0.74 | flip-aav-seven-vs-rest-spearman |
| Levenshtein distance baseline (FLIP) | 0.53 | flip-aav-seven-vs-rest-spearman |
Spearman correlation
flip-aav-two-vs-rest · flip-aav-two-vs-rest-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.65 | flip-aav-two-vs-rest-spearman |
| ESM-1b (mean; FLIP head) | 0.26 | flip-aav-two-vs-rest-spearman |
| ESM-1b (mut mean; FLIP head) | 0.65 | flip-aav-two-vs-rest-spearman |
| ESM-1v (per AA; FLIP head) | 0.7 | flip-aav-two-vs-rest-spearman |
| ESM-1v (mean; FLIP head) | 0.16 | flip-aav-two-vs-rest-spearman |
| ESM-1v (mut mean; FLIP head) | 0.64 | flip-aav-two-vs-rest-spearman |
| ESM-untrained (per AA) | 0.22 | flip-aav-two-vs-rest-spearman |
| ESM-untrained (mean) | 0.14 | flip-aav-two-vs-rest-spearman |
| ESM-untrained (mut mean) | 0.16 | flip-aav-two-vs-rest-spearman |
| Ridge regression (FLIP) | 0.03 | flip-aav-two-vs-rest-spearman |
| Convolutional network (FLIP) | 0.74 | flip-aav-two-vs-rest-spearman |
| Levenshtein distance baseline (FLIP) | 0.57 | flip-aav-two-vs-rest-spearman |
Spearman correlation
flip-gb1-low-vs-high · flip-gb1-low-vs-high-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.59 | flip-gb1-low-vs-high-spearman |
| ESM-1b (mean; FLIP head) | 0.13 | flip-gb1-low-vs-high-spearman |
| ESM-1b (mut mean; FLIP head) | 0.45 | flip-gb1-low-vs-high-spearman |
| ESM-1v (per AA; FLIP head) | 0.51 | flip-gb1-low-vs-high-spearman |
| ESM-1v (mean; FLIP head) | 0.1 | flip-gb1-low-vs-high-spearman |
| ESM-1v (mut mean; FLIP head) | 0.49 | flip-gb1-low-vs-high-spearman |
| ESM-untrained (per AA) | 0.23 | flip-gb1-low-vs-high-spearman |
| ESM-untrained (mean) | 0.1 | flip-gb1-low-vs-high-spearman |
| ESM-untrained (mut mean) | 0.13 | flip-gb1-low-vs-high-spearman |
| Ridge regression (FLIP) | 0.34 | flip-gb1-low-vs-high-spearman |
| Convolutional network (FLIP) | 0.51 | flip-gb1-low-vs-high-spearman |
| Levenshtein distance baseline (FLIP) | -0.1 | flip-gb1-low-vs-high-spearman |
| BLOSUM62 baseline (FLIP) | -0.13 | flip-gb1-low-vs-high-spearman |
Spearman correlation
flip-gb1-one-vs-rest · flip-gb1-one-vs-rest-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.28 | flip-gb1-one-vs-rest-spearman |
| ESM-1b (mean; FLIP head) | 0.32 | flip-gb1-one-vs-rest-spearman |
| ESM-1b (mut mean; FLIP head) | -0.08 | flip-gb1-one-vs-rest-spearman |
| ESM-1v (per AA; FLIP head) | 0.28 | flip-gb1-one-vs-rest-spearman |
| ESM-1v (mean; FLIP head) | 0.32 | flip-gb1-one-vs-rest-spearman |
| ESM-1v (mut mean; FLIP head) | 0.19 | flip-gb1-one-vs-rest-spearman |
| ESM-untrained (per AA) | 0.06 | flip-gb1-one-vs-rest-spearman |
| ESM-untrained (mean) | 0.05 | flip-gb1-one-vs-rest-spearman |
| ESM-untrained (mut mean) | 0.21 | flip-gb1-one-vs-rest-spearman |
| Ridge regression (FLIP) | 0.28 | flip-gb1-one-vs-rest-spearman |
| Convolutional network (FLIP) | 0.17 | flip-gb1-one-vs-rest-spearman |
| Levenshtein distance baseline (FLIP) | 0.17 | flip-gb1-one-vs-rest-spearman |
| BLOSUM62 baseline (FLIP) | 0.15 | flip-gb1-one-vs-rest-spearman |
Spearman correlation
flip-gb1-sampled · flip-gb1-sampled-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.92 | flip-gb1-sampled-spearman |
| ESM-1v (per AA; FLIP head) | 0.92 | flip-gb1-sampled-spearman |
| ESM-untrained (per AA) | 0.79 | flip-gb1-sampled-spearman |
| Ridge regression (FLIP) | 0.82 | flip-gb1-sampled-spearman |
| Convolutional network (FLIP) | 0.91 | flip-gb1-sampled-spearman |
Spearman correlation
flip-gb1-three-vs-rest · flip-gb1-three-vs-rest-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.79 | flip-gb1-three-vs-rest-spearman |
| ESM-1b (mean; FLIP head) | 0.54 | flip-gb1-three-vs-rest-spearman |
| ESM-1b (mut mean; FLIP head) | 0.49 | flip-gb1-three-vs-rest-spearman |
| ESM-1v (per AA; FLIP head) | 0.82 | flip-gb1-three-vs-rest-spearman |
| ESM-1v (mean; FLIP head) | 0.77 | flip-gb1-three-vs-rest-spearman |
| ESM-1v (mut mean; FLIP head) | 0.8 | flip-gb1-three-vs-rest-spearman |
| ESM-untrained (per AA) | 0.48 | flip-gb1-three-vs-rest-spearman |
| ESM-untrained (mean) | 0.46 | flip-gb1-three-vs-rest-spearman |
| ESM-untrained (mut mean) | 0.57 | flip-gb1-three-vs-rest-spearman |
| Ridge regression (FLIP) | 0.76 | flip-gb1-three-vs-rest-spearman |
| Convolutional network (FLIP) | 0.83 | flip-gb1-three-vs-rest-spearman |
| Levenshtein distance baseline (FLIP) | -0.04 | flip-gb1-three-vs-rest-spearman |
| BLOSUM62 baseline (FLIP) | 0.01 | flip-gb1-three-vs-rest-spearman |
Spearman correlation
flip-gb1-two-vs-rest · flip-gb1-two-vs-rest-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.55 | flip-gb1-two-vs-rest-spearman |
| ESM-1b (mean; FLIP head) | 0.36 | flip-gb1-two-vs-rest-spearman |
| ESM-1b (mut mean; FLIP head) | 0.19 | flip-gb1-two-vs-rest-spearman |
| ESM-1v (per AA; FLIP head) | 0.28 | flip-gb1-two-vs-rest-spearman |
| ESM-1v (mean; FLIP head) | 0.32 | flip-gb1-two-vs-rest-spearman |
| ESM-1v (mut mean; FLIP head) | 0.19 | flip-gb1-two-vs-rest-spearman |
| ESM-untrained (per AA) | 0.06 | flip-gb1-two-vs-rest-spearman |
| ESM-untrained (mean) | 0.05 | flip-gb1-two-vs-rest-spearman |
| ESM-untrained (mut mean) | 0.21 | flip-gb1-two-vs-rest-spearman |
| Ridge regression (FLIP) | 0.59 | flip-gb1-two-vs-rest-spearman |
| Convolutional network (FLIP) | 0.32 | flip-gb1-two-vs-rest-spearman |
| Levenshtein distance baseline (FLIP) | 0.16 | flip-gb1-two-vs-rest-spearman |
| BLOSUM62 baseline (FLIP) | 0.14 | flip-gb1-two-vs-rest-spearman |
Spearman correlation
flip-meltome-human · flip-meltome-human-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.71 | flip-meltome-human-spearman |
| ESM-1b (mean; FLIP head) | 0.7 | flip-meltome-human-spearman |
| ESM-1v (per AA; FLIP head) | 0.77 | flip-meltome-human-spearman |
| ESM-1v (mean; FLIP head) | 0.75 | flip-meltome-human-spearman |
| ESM-untrained (per AA) | 0.44 | flip-meltome-human-spearman |
| ESM-untrained (mean) | 0.48 | flip-meltome-human-spearman |
| Ridge regression (FLIP) | 0.15 | flip-meltome-human-spearman |
| Convolutional network (FLIP) | 0.5 | flip-meltome-human-spearman |
Spearman correlation
flip-meltome-human-cell · flip-meltome-human-cell-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.76 | flip-meltome-human-cell-spearman |
| ESM-1b (mean; FLIP head) | 0.75 | flip-meltome-human-cell-spearman |
| ESM-1v (per AA; FLIP head) | 0.78 | flip-meltome-human-cell-spearman |
| ESM-1v (mean; FLIP head) | 0.74 | flip-meltome-human-cell-spearman |
| ESM-untrained (per AA) | 0.46 | flip-meltome-human-cell-spearman |
| ESM-untrained (mean) | 0.49 | flip-meltome-human-cell-spearman |
| Ridge regression (FLIP) | 0.24 | flip-meltome-human-cell-spearman |
| Convolutional network (FLIP) | 0.49 | flip-meltome-human-cell-spearman |
Spearman correlation
flip-meltome-mixed · flip-meltome-mixed-spearman
Accessible data table
| Model | Value | Comparability group |
|---|---|---|
| ESM-1b (per AA; FLIP head) | 0.68 | flip-meltome-mixed-spearman |
| ESM-1b (mean; FLIP head) | 0.68 | flip-meltome-mixed-spearman |
| ESM-1v (per AA; FLIP head) | 0.65 | flip-meltome-mixed-spearman |
| ESM-1v (mean; FLIP head) | 0.67 | flip-meltome-mixed-spearman |
| ESM-untrained (per AA) | 0.44 | flip-meltome-mixed-spearman |
| ESM-untrained (mean) | 0.36 | flip-meltome-mixed-spearman |
| Ridge regression (FLIP) | 0.17 | flip-meltome-mixed-spearman |
| Convolutional network (FLIP) | 0.34 | flip-meltome-mixed-spearman |
Evidence and change history
Source locators remain visible; expand an item to inspect the exact Registry fields it supports.
FLIP: Benchmark tasks in fitness landscape inference for proteins · table: pp. 3–7, Figure 1 and Tables 2–3 (Defines the three landscapes, 15 dataset splits, task inputs/outputs, baseline classes, sample counts, licenses, and protein-engineering motivation.) · Supports 26 fields
/name/aliases/summary/kind/organizations/release_date/domains/capabilities/modalities/task_formats/task_counts/total/task_counts/basis/task_counts/subsets/coverage_notes/access/level/access/tasks/access/artifacts/access/grader/access/license/access/biosafety_notes/versions/0/release_date/versions/0/task_counts/total/versions/0/task_counts/basis/versions/0/task_counts/subsets/scientific_task_classification/entries/0/scientific_task_classification/entries/1
flip-repository-resource · repository-path: README.md; splits/README.md; splits/{aav,gb1,meltome}/README.md; LICENSE at commit 62cace8735f5610e2743cf06ce0f944b37fffaa6 (Separates active, discourse-only, and obsolete splits; documents dataset-specific licensing and pins the runnable implementation.) · Supports 7 fields
/latest_version/task_counts/subsets/resources/implementations/versions/0/as_of/versions/0/task_counts/subsets/versions/0/formal_tracks