Benchmarks/FLIP/FLIP Meltome Thermostability

track · audited · verified 2026-07-21

FLIP Meltome Thermostability

Three supervised sequence-to-melting-temperature splits spanning all species, human proteins, and a single human cell line, with sequence-cluster-aware train/test separation.

Human-cell count resolution: creator-paper Table 2 prints 7,156 total examples, although its 5,792 train and 1,366 test counts sum to 7,158. The pinned official CSV contains 7,158 rows, which is the registered value.

Benchmark definition

What is counted

Version: original-2021
Total: 27951 (sequence-temperature examples in the Mixed split universe)
Task formats: sequence-to-melting-temperature regression
Capabilities: PredictionRegression
Modalities: Protein sequenceTable

Version history

Version	Status	Release / as-of	Total	Formal tracks
original-2021 `flip-meltome-original-2021`	current	2021-10-11	27951 (sequence-temperature examples in the Mixed split universe)	None registered

Tracks and subsets

ID	Count	Basis	Partition?	Notes
Mixed test set `meltome-mixed-test`	3134	test examples	No	24,817 training examples; test contains representatives from held-out 20%-identity clusters.
Human split total `meltome-human-all`	10093	sequence-temperature examples	No	Human-only subset of the landscape.
Human test set `meltome-human-test`	1945	test examples	No	8,148 training examples.
Human-cell split total `meltome-human-cell-all`	7158	sequence-temperature examples	No	Pinned official CSV has 7,158 rows (5,792 train + 1,366 test); paper Table 2 prints 7,156.
Human-cell test set `meltome-human-cell-test`	1366	test examples	No	5,792 training examples in the pinned CSV and paper.

Scientific Task Atlas

Scientific task classification

complete for original-2021. Protein melting-temperature landscape.

Scientific task	Coverage	Count	Mapping	Evidence
Protein stability prediction	explicitly-in-scope	27951 examples Sequence-temperature examples in the Mixed split universe.	official-track high confidence	`flip-meltome-evidence-paper` Temperature examples are not combined with AAV or GB1 counts.

Scientific coverage notes

Domain	Coverage	Count	Interpretation
Proteomics	explicitly-in-scope	27951	The benchmark uses Meltome Atlas thermal-proteome measurements; count is the Mixed split universe, not independent assays.

Evaluation registry

Works and run settings

A setting change—scope, prompt, tools, budget, grader, or repeats—creates a separate run. Charts never cross a comparability group.

Evaluation run

flip-meltome-human

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Evaluated models / systems: Convolutional network (FLIP), ESM-untrained (mean), ESM-untrained (per AA), ESM-1b (mean; FLIP head), ESM-1b (per AA; FLIP head), ESM-1v (mean; FLIP head), ESM-1v (per AA; FLIP head), Ridge regression (FLIP)

Scopesubset · n=1945

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationHuman-only sequence-cluster holdout at 20% identity.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.71 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-1b (mean; FLIP head)	Spearman correlation	0.7 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-1v (per AA; FLIP head)	Spearman correlation	0.77 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-1v (mean; FLIP head)	Spearman correlation	0.75 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-untrained (per AA)	Spearman correlation	0.44 correlation Held-out test-set Spearman from the creator paper.	1945
ESM-untrained (mean)	Spearman correlation	0.48 correlation Held-out test-set Spearman from the creator paper.	1945
Ridge regression (FLIP)	Spearman correlation	0.15 correlation Held-out test-set Spearman from the creator paper.	1945
Convolutional network (FLIP)	Spearman correlation	0.5 correlation Held-out test-set Spearman from the creator paper.	1945

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 6 (Human column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-meltome-human-cell

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=1366

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationSingle-cell-line sequence-cluster holdout at 20% identity.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.76 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-1b (mean; FLIP head)	Spearman correlation	0.75 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-1v (per AA; FLIP head)	Spearman correlation	0.78 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-1v (mean; FLIP head)	Spearman correlation	0.74 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-untrained (per AA)	Spearman correlation	0.46 correlation Held-out test-set Spearman from the creator paper.	1366
ESM-untrained (mean)	Spearman correlation	0.49 correlation Held-out test-set Spearman from the creator paper.	1366
Ridge regression (FLIP)	Spearman correlation	0.24 correlation Held-out test-set Spearman from the creator paper.	1366
Convolutional network (FLIP)	Spearman correlation	0.49 correlation Held-out test-set Spearman from the creator paper.	1366

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 6 (Human-Cell column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Evaluation run

flip-meltome-mixed

From FLIP: Benchmark tasks in fitness landscape inference for proteins

Scopesubset · n=3134

ShotsNot applicable

TurnsNot applicable

System prompt publicNot applicable

Reasoning / effortNot applicable

BrowserNot applicable

InternetNot applicable

DatabasesNot applicable

Code executionNot applicable

ContainerNot reported

External toolsmodel-specific frozen embeddings or one-hot sequence encoding

Token budgetNot applicable

Time / cost budgetNot reported

TemperatureNot applicable

SeedNot reported

RepeatsNot reported

Graderdeterministic regression scorer · human review: no

StatisticsSpearman correlation over held-out test examples; the main table reports point values without confidence intervals.

ContaminationSequence-cluster holdout at 20% identity; test uses held-out cluster representatives.

Metrics, results, and full protocol

Metrics

Metric	Kind / baseline	Unit	Aggregation	Threshold / tolerance
Spearman correlation	absolute	correlation	computed across all examples in this held-out test split	Not reported
Mean squared error	absolute	squared fitness units	mean across all examples in this held-out test split	Not reported

Results

Model	Metric	Value	n
ESM-1b (per AA; FLIP head)	Spearman correlation	0.68 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-1b (mean; FLIP head)	Spearman correlation	0.68 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-1v (per AA; FLIP head)	Spearman correlation	0.65 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-1v (mean; FLIP head)	Spearman correlation	0.67 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-untrained (per AA)	Spearman correlation	0.44 correlation Held-out test-set Spearman from the creator paper.	3134
ESM-untrained (mean)	Spearman correlation	0.36 correlation Held-out test-set Spearman from the creator paper.	3134
Ridge regression (FLIP)	Spearman correlation	0.17 correlation Held-out test-set Spearman from the creator paper.	3134
Convolutional network (FLIP)	Spearman correlation	0.34 correlation Held-out test-set Spearman from the creator paper.	3134

Evidence

table: pp. 4–7, Table 2 and Sections 3–4 (Exact train/test counts, split rule, baseline identities, pooling, optimizer, batch sizes, early stopping, and hardware.) — supports /scope, /protocol, /model_ids
table: p. 9, Table 6 (Mixed column) (Published held-out Spearman values; em dashes and NA entries are omitted rather than encoded as zeros.) — supports /metrics, /results

Comparable result views

Spearman correlation

flip-meltome-human · flip-meltome-human-spearman

CSV ↓

Accessible data table

Model	Value	Comparability group
ESM-1b (per AA; FLIP head)	0.71	flip-meltome-human-spearman
ESM-1b (mean; FLIP head)	0.7	flip-meltome-human-spearman
ESM-1v (per AA; FLIP head)	0.77	flip-meltome-human-spearman
ESM-1v (mean; FLIP head)	0.75	flip-meltome-human-spearman
ESM-untrained (per AA)	0.44	flip-meltome-human-spearman
ESM-untrained (mean)	0.48	flip-meltome-human-spearman
Ridge regression (FLIP)	0.15	flip-meltome-human-spearman
Convolutional network (FLIP)	0.5	flip-meltome-human-spearman

Spearman correlation

flip-meltome-human-cell · flip-meltome-human-cell-spearman

CSV ↓

Accessible data table

Model	Value	Comparability group
ESM-1b (per AA; FLIP head)	0.76	flip-meltome-human-cell-spearman
ESM-1b (mean; FLIP head)	0.75	flip-meltome-human-cell-spearman
ESM-1v (per AA; FLIP head)	0.78	flip-meltome-human-cell-spearman
ESM-1v (mean; FLIP head)	0.74	flip-meltome-human-cell-spearman
ESM-untrained (per AA)	0.46	flip-meltome-human-cell-spearman
ESM-untrained (mean)	0.49	flip-meltome-human-cell-spearman
Ridge regression (FLIP)	0.24	flip-meltome-human-cell-spearman
Convolutional network (FLIP)	0.49	flip-meltome-human-cell-spearman

Spearman correlation

flip-meltome-mixed · flip-meltome-mixed-spearman

CSV ↓

Accessible data table

Model	Value	Comparability group
ESM-1b (per AA; FLIP head)	0.68	flip-meltome-mixed-spearman
ESM-1b (mean; FLIP head)	0.68	flip-meltome-mixed-spearman
ESM-1v (per AA; FLIP head)	0.65	flip-meltome-mixed-spearman
ESM-1v (mean; FLIP head)	0.67	flip-meltome-mixed-spearman
ESM-untrained (per AA)	0.44	flip-meltome-mixed-spearman
ESM-untrained (mean)	0.36	flip-meltome-mixed-spearman
Ridge regression (FLIP)	0.17	flip-meltome-mixed-spearman
Convolutional network (FLIP)	0.34	flip-meltome-mixed-spearman

Evidence and change history

Source locators remain visible; expand an item to inspect the exact Registry fields it supports.

FLIP: Benchmark tasks in fitness landscape inference for proteins · table: pp. 4, 6, 9; Table 2, Section 3.3, Table 6 (Defines three splits, train/test counts, 20%-identity clustering, metric, and baseline results; Table 2 prints Human-cell total 7,156.) · Supports 26 fields

Open source →

/name
/summary
/kind
/organizations
/release_date
/latest_version
/domains
/capabilities
/modalities
/task_formats
/task_counts/total
/task_counts/basis
/task_counts/subsets
/coverage_notes
/access/level
/access/tasks
/access/artifacts
/access/grader
/access/license
/access/biosafety_notes
/versions/0/release_date
/versions/0/as_of
/versions/0/task_counts/total
/versions/0/task_counts/basis
/versions/0/task_counts/subsets
/scientific_task_classification/entries/0

flip-meltome-repository-resource · repository-path: splits/meltome/README.md and splits/meltome/splits.zip at commit 62cace8735f5610e2743cf06ce0f944b37fffaa6 (CSV counts: Mixed 27,951=24,817+3,134; Human 10,093=8,148+1,945; Human-cell 7,158=5,792+1,366.) · Supports 7 fields

Open source →

/task_counts/subsets
/access/tasks
/access/artifacts
/access/license
/resources
/implementations
/versions/0/task_counts/subsets

View source-level modification history on GitHub →

Access & reproducibility

Access: Fully open
License: Meltome creators state the original data are free to use with acknowledgement; FLIP-modified task files are AFL-3.0.
Grader: Deterministic Spearman correlation and MSE.
Implementations: 1 recorded · official