paper · benchmark creator

FLIP: Benchmark tasks in fitness landscape inference for proteins

Technical University of Munich · Microsoft Research New England · California Institute of Technology · University of California Berkeley · Massachusetts Institute of Technology · Salesforce Research · 2021-10-11

Relationship layer

Benchmark usage

This table records what the work did with each benchmark before attempting to normalize a run. Partial claims remain visible without being treated as comparable evaluations.

No BenchmarkUse relation is normalized for this legacy work yet. Existing EvaluationRuns remain available below.

Normalized evaluation runs

FLIP AAV7 runs

Open benchmark record →

Evaluation run

flip-aav-des-mut

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)

Scopesubset · n=82583

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationPool shift: model-designed variants are training data and sampled variants are held out.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (mean; FLIP head)	Spearman correlation	0.59 correlation Held-out test-set Spearman from the creator paper.	82583
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	82583
ESM-1v (mean; FLIP head)	Spearman correlation	0.44 correlation Held-out test-set Spearman from the creator paper.	82583
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.71 correlation Held-out test-set Spearman from the creator paper.	82583
ESM-untrained (mean)	Spearman correlation	0.34 correlation Held-out test-set Spearman from the creator paper.	82583
ESM-untrained (mut mean)	Spearman correlation	0.64 correlation Held-out test-set Spearman from the creator paper.	82583
Ridge regression (FLIP)	Spearman correlation	0.53 correlation Held-out test-set Spearman from the creator paper.	82583
Convolutional network (FLIP)	Spearman correlation	0.75 correlation Held-out test-set Spearman from the creator paper.	82583
Levenshtein distance baseline (FLIP)	Spearman correlation	-0.07 correlation Held-out test-set Spearman from the creator paper.	82583

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 5 (Des-Mut column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-aav-low-vs-high

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)

Scopesubset · n=35037

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationFitness extrapolation above wild-type fitness.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.39 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-1b (mean; FLIP head)	Spearman correlation	0.18 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.33 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-1v (per AA; FLIP head)	Spearman correlation	0.34 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-1v (mean; FLIP head)	Spearman correlation	0.2 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.31 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-untrained (per AA)	Spearman correlation	0.08 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-untrained (mean)	Spearman correlation	0.22 correlation Held-out test-set Spearman from the creator paper.	35037
ESM-untrained (mut mean)	Spearman correlation	0.24 correlation Held-out test-set Spearman from the creator paper.	35037
Ridge regression (FLIP)	Spearman correlation	0.12 correlation Held-out test-set Spearman from the creator paper.	35037
Convolutional network (FLIP)	Spearman correlation	0.34 correlation Held-out test-set Spearman from the creator paper.	35037
Levenshtein distance baseline (FLIP)	Spearman correlation	0.25 correlation Held-out test-set Spearman from the creator paper.	35037

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 5 (low-vs-high column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-aav-mut-des

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=201426

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationPool shift: sampled variants are training data and model-designed variants are held out.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.76 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-1b (mean; FLIP head)	Spearman correlation	0.63 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-1v (per AA; FLIP head)	Spearman correlation	0.79 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-1v (mean; FLIP head)	Spearman correlation	0.55 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-untrained (per AA)	Spearman correlation	0.56 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-untrained (mean)	Spearman correlation	0.27 correlation Held-out test-set Spearman from the creator paper.	201426
ESM-untrained (mut mean)	Spearman correlation	0.62 correlation Held-out test-set Spearman from the creator paper.	201426
Ridge regression (FLIP)	Spearman correlation	0.64 correlation Held-out test-set Spearman from the creator paper.	201426
Convolutional network (FLIP)	Spearman correlation	0.71 correlation Held-out test-set Spearman from the creator paper.	201426
Levenshtein distance baseline (FLIP)	Spearman correlation	0.6 correlation Held-out test-set Spearman from the creator paper.	201426

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 5 (Mut-Des column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-aav-one-vs-rest

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=81413

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationMutation-depth extrapolation; the held-out set excludes wild type and single-mutant training examples.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.03 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-1b (mean; FLIP head)	Spearman correlation	0.04 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.31 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-1v (per AA; FLIP head)	Spearman correlation	0.1 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-1v (mean; FLIP head)	Spearman correlation	0.18 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.44 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-untrained (per AA)	Spearman correlation	0.18 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-untrained (mean)	Spearman correlation	0.01 correlation Held-out test-set Spearman from the creator paper.	81413
ESM-untrained (mut mean)	Spearman correlation	0.26 correlation Held-out test-set Spearman from the creator paper.	81413
Ridge regression (FLIP)	Spearman correlation	0.22 correlation Held-out test-set Spearman from the creator paper.	81413
Convolutional network (FLIP)	Spearman correlation	0.48 correlation Held-out test-set Spearman from the creator paper.	81413
Levenshtein distance baseline (FLIP)	Spearman correlation	-0.11 correlation Held-out test-set Spearman from the creator paper.	81413

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 5 (1-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-aav-sampled

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (per AA), ESM-1b (per AA; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)

Scopesubset · n=16517

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationRandom split can place closely related variants across train/test and is explicitly described as optimistic.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.9 correlation Held-out test-set Spearman from the creator paper.	16517
ESM-1v (per AA; FLIP head)	Spearman correlation	0.92 correlation Held-out test-set Spearman from the creator paper.	16517
ESM-untrained (per AA)	Spearman correlation	0.78 correlation Held-out test-set Spearman from the creator paper.	16517
Ridge regression (FLIP)	Spearman correlation	0.83 correlation Held-out test-set Spearman from the creator paper.	16517
Convolutional network (FLIP)	Spearman correlation	0.92 correlation Held-out test-set Spearman from the creator paper.	16517

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 10, Table 7 (AAV column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-aav-seven-vs-rest

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=12581

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationMutation-depth extrapolation above seven changes.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.65 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-1b (mean; FLIP head)	Spearman correlation	0.46 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.61 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-1v (per AA; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-1v (mean; FLIP head)	Spearman correlation	0.45 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.64 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-untrained (per AA)	Spearman correlation	0.42 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-untrained (mean)	Spearman correlation	0.22 correlation Held-out test-set Spearman from the creator paper.	12581
ESM-untrained (mut mean)	Spearman correlation	0.56 correlation Held-out test-set Spearman from the creator paper.	12581
Ridge regression (FLIP)	Spearman correlation	0.65 correlation Held-out test-set Spearman from the creator paper.	12581
Convolutional network (FLIP)	Spearman correlation	0.74 correlation Held-out test-set Spearman from the creator paper.	12581
Levenshtein distance baseline (FLIP)	Spearman correlation	0.53 correlation Held-out test-set Spearman from the creator paper.	12581

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 5 (7-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-aav-two-vs-rest

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=50776

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationMutation-depth extrapolation above two changes.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.65 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-1b (mean; FLIP head)	Spearman correlation	0.26 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.65 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-1v (per AA; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-1v (mean; FLIP head)	Spearman correlation	0.16 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.64 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-untrained (per AA)	Spearman correlation	0.22 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-untrained (mean)	Spearman correlation	0.14 correlation Held-out test-set Spearman from the creator paper.	50776
ESM-untrained (mut mean)	Spearman correlation	0.16 correlation Held-out test-set Spearman from the creator paper.	50776
Ridge regression (FLIP)	Spearman correlation	0.03 correlation Held-out test-set Spearman from the creator paper.	50776
Convolutional network (FLIP)	Spearman correlation	0.74 correlation Held-out test-set Spearman from the creator paper.	50776
Levenshtein distance baseline (FLIP)	Spearman correlation	0.57 correlation Held-out test-set Spearman from the creator paper.	50776

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 5 (2-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

FLIP GB15 runs

Open benchmark record →

Evaluation run

flip-gb1-low-vs-high

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: BLOSUM62 baseline (FLIP), Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (mut mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (mut mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (mut mean; FLIP head), ESM-1v (per AA; FLIP head), Levenshtein distance baseline (FLIP), Ridge regression (FLIP)

Scopesubset · n=3644

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationFitness extrapolation above wild-type fitness.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.59 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-1b (mean; FLIP head)	Spearman correlation	0.13 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.45 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-1v (per AA; FLIP head)	Spearman correlation	0.51 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-1v (mean; FLIP head)	Spearman correlation	0.1 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.49 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-untrained (per AA)	Spearman correlation	0.23 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-untrained (mean)	Spearman correlation	0.1 correlation Held-out test-set Spearman from the creator paper.	3644
ESM-untrained (mut mean)	Spearman correlation	0.13 correlation Held-out test-set Spearman from the creator paper.	3644
Ridge regression (FLIP)	Spearman correlation	0.34 correlation Held-out test-set Spearman from the creator paper.	3644
Convolutional network (FLIP)	Spearman correlation	0.51 correlation Held-out test-set Spearman from the creator paper.	3644
Levenshtein distance baseline (FLIP)	Spearman correlation	-0.1 correlation Held-out test-set Spearman from the creator paper.	3644
BLOSUM62 baseline (FLIP)	Spearman correlation	-0.13 correlation Held-out test-set Spearman from the creator paper.	3644

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 8, Table 4 (low-vs-high column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-gb1-one-vs-rest

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=8704

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationMutation-depth extrapolation from wild type and single mutants.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.28 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-1b (mean; FLIP head)	Spearman correlation	0.32 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-1b (mut mean; FLIP head)	Spearman correlation	-0.08 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-1v (per AA; FLIP head)	Spearman correlation	0.28 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-1v (mean; FLIP head)	Spearman correlation	0.32 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.19 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-untrained (per AA)	Spearman correlation	0.06 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-untrained (mean)	Spearman correlation	0.05 correlation Held-out test-set Spearman from the creator paper.	8704
ESM-untrained (mut mean)	Spearman correlation	0.21 correlation Held-out test-set Spearman from the creator paper.	8704
Ridge regression (FLIP)	Spearman correlation	0.28 correlation Held-out test-set Spearman from the creator paper.	8704
Convolutional network (FLIP)	Spearman correlation	0.17 correlation Held-out test-set Spearman from the creator paper.	8704
Levenshtein distance baseline (FLIP)	Spearman correlation	0.17 correlation Held-out test-set Spearman from the creator paper.	8704
BLOSUM62 baseline (FLIP)	Spearman correlation	0.15 correlation Held-out test-set Spearman from the creator paper.	8704

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 8, Table 4 (1-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-gb1-sampled

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (per AA), ESM-1b (per AA; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)

Scopesubset · n=1772

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationRandom split can place closely related variants across train/test and is explicitly described as optimistic.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.92 correlation Held-out test-set Spearman from the creator paper.	1772
ESM-1v (per AA; FLIP head)	Spearman correlation	0.92 correlation Held-out test-set Spearman from the creator paper.	1772
ESM-untrained (per AA)	Spearman correlation	0.79 correlation Held-out test-set Spearman from the creator paper.	1772
Ridge regression (FLIP)	Spearman correlation	0.82 correlation Held-out test-set Spearman from the creator paper.	1772
Convolutional network (FLIP)	Spearman correlation	0.91 correlation Held-out test-set Spearman from the creator paper.	1772

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 10, Table 7 (GB1 column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-gb1-three-vs-rest

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=5765

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationMutation-depth extrapolation to four simultaneous mutations.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.79 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-1b (mean; FLIP head)	Spearman correlation	0.54 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.49 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-1v (per AA; FLIP head)	Spearman correlation	0.82 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-1v (mean; FLIP head)	Spearman correlation	0.77 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.8 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-untrained (per AA)	Spearman correlation	0.48 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-untrained (mean)	Spearman correlation	0.46 correlation Held-out test-set Spearman from the creator paper.	5765
ESM-untrained (mut mean)	Spearman correlation	0.57 correlation Held-out test-set Spearman from the creator paper.	5765
Ridge regression (FLIP)	Spearman correlation	0.76 correlation Held-out test-set Spearman from the creator paper.	5765
Convolutional network (FLIP)	Spearman correlation	0.83 correlation Held-out test-set Spearman from the creator paper.	5765
Levenshtein distance baseline (FLIP)	Spearman correlation	-0.04 correlation Held-out test-set Spearman from the creator paper.	5765
BLOSUM62 baseline (FLIP)	Spearman correlation	0.01 correlation Held-out test-set Spearman from the creator paper.	5765

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 8, Table 4 (3-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-gb1-two-vs-rest

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=8306

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationMutation-depth extrapolation above two mutations.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.55 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-1b (mean; FLIP head)	Spearman correlation	0.36 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-1b (mut mean; FLIP head)	Spearman correlation	0.19 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-1v (per AA; FLIP head)	Spearman correlation	0.28 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-1v (mean; FLIP head)	Spearman correlation	0.32 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-1v (mut mean; FLIP head)	Spearman correlation	0.19 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-untrained (per AA)	Spearman correlation	0.06 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-untrained (mean)	Spearman correlation	0.05 correlation Held-out test-set Spearman from the creator paper.	8306
ESM-untrained (mut mean)	Spearman correlation	0.21 correlation Held-out test-set Spearman from the creator paper.	8306
Ridge regression (FLIP)	Spearman correlation	0.59 correlation Held-out test-set Spearman from the creator paper.	8306
Convolutional network (FLIP)	Spearman correlation	0.32 correlation Held-out test-set Spearman from the creator paper.	8306
Levenshtein distance baseline (FLIP)	Spearman correlation	0.16 correlation Held-out test-set Spearman from the creator paper.	8306
BLOSUM62 baseline (FLIP)	Spearman correlation	0.14 correlation Held-out test-set Spearman from the creator paper.	8306

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 8, Table 4 (2-vs-rest column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

FLIP Meltome Thermostability3 runs

Open benchmark record →

Evaluation run

flip-meltome-human

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)

Scopesubset · n=1945

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationHuman-only sequence-cluster holdout at 20% identity.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.71 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-1b (mean; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-1v (per AA; FLIP head)	Spearman correlation	0.77 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-1v (mean; FLIP head)	Spearman correlation	0.75 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-untrained (per AA)	Spearman correlation	0.44 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-untrained (mean)	Spearman correlation	0.48 correlation Held-out test-set Spearman from the creator paper.	1945
Ridge regression (FLIP)	Spearman correlation	0.15 correlation Held-out test-set Spearman from the creator paper.	1945
Convolutional network (FLIP)	Spearman correlation	0.5 correlation Held-out test-set Spearman from the creator paper.	1945

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 6 (Human column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-meltome-human-cell

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=1366

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationSingle-cell-line sequence-cluster holdout at 20% identity.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.76 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-1b (mean; FLIP head)	Spearman correlation	0.75 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-1v (per AA; FLIP head)	Spearman correlation	0.78 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-1v (mean; FLIP head)	Spearman correlation	0.74 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-untrained (per AA)	Spearman correlation	0.46 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-untrained (mean)	Spearman correlation	0.49 correlation Held-out test-set Spearman from the creator paper.	1366
Ridge regression (FLIP)	Spearman correlation	0.24 correlation Held-out test-set Spearman from the creator paper.	1366
Convolutional network (FLIP)	Spearman correlation	0.49 correlation Held-out test-set Spearman from the creator paper.	1366

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 6 (Human-Cell column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-meltome-mixed

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=3134

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationSequence-cluster holdout at 20% identity; test uses held-out cluster representatives.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.68 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-1b (mean; FLIP head)	Spearman correlation	0.68 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-1v (per AA; FLIP head)	Spearman correlation	0.65 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-1v (mean; FLIP head)	Spearman correlation	0.67 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-untrained (per AA)	Spearman correlation	0.44 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-untrained (mean)	Spearman correlation	0.36 correlation Held-out test-set Spearman from the creator paper.	3134
Ridge regression (FLIP)	Spearman correlation	0.17 correlation Held-out test-set Spearman from the creator paper.	3134
Convolutional network (FLIP)	Spearman correlation	0.34 correlation Held-out test-set Spearman from the creator paper.	3134

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 6 (Mixed column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results