Published on Mar 10, 2025 by Arcadia Science Paired residue prediction dependencies in ESM2
During a quick analysis of the ESM2 model for masked token prediction, we noticed that amino acid probability distributions of residues affect each other in a pattern that mirrors a protein’s 3D contact map. But less so for the larger model sizes. Our question to you is, why?
Contributors (A-Z)
Version 3 · Dec 03, 2025
Paired residue prediction dependencies in ESM2
Purpose
This notebook details a quick analysis we performed on the ESM2 (Evolutionary Scale Modeling) models [1] for masked token prediction. We stumbled upon a counterintuitive result related to the effect that masking one residue has on the distribution of another.
Before jumping into our results, let’s first establish what masked token prediction is, how it works, and our consequent motivation for this analysis.
The full notebook pub is available here.
The source code to generate it is available in this GitHub repo (DOI: 10.5281/zenodo.15002836).
References
1
Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W, Smetanin N, Verkuil R, Kabeli O, Shmueli Y, dos Santos Costa A, Fazel-Zarandi M, Sercu T, Candido S, Rives A. (2023). Evolutionary-scale prediction of atomic-level protein structure with a language model. https://doi.org/10.1126/science.ade2574
Pub details
Arcadia
Bigge BM, Kiefl E. (2025). Paired residue prediction dependencies in ESM2. https://doi.org/10.57844/arcadia-f52b-1451
apa
Bigge, B. M., & Kiefl, E. (2025). Paired residue prediction dependencies in ESM2. Arcadia Science. https://doi.org/10.57844/arcadia-f52b-1451
harvard
Bigge, B.M. and Kiefl, E. (2025) 'Paired residue prediction dependencies in ESM2', Arcadia Science [Preprint]. doi:10.57844/arcadia-f52b-1451.
vancouver
1.
Bigge BM, Kiefl E. Paired residue prediction dependencies in ESM2. Arcadia Science [Internet]. 2025 Mar 10; Available from: https://research.arcadiascience.com/pub/result-paired-token-masking/release/3
bibtex
@article{Bigge2025Paired,
author = {Bigge, Brae M. and Kiefl, Evan},
journal = {Arcadia Science},
doi = {10.57844/arcadia-f52b-1451},
issn = {2998-4084},
year = {2025},
month = {mar 10},
publisher = {Arcadia Science},
title = {Paired residue prediction dependencies in {ESM2}},
url = {https://research.arcadiascience.com/pub/result-paired-token-masking/release/3},
}
Brae M. Bigge
Supervision
Daniel Burns
Critical Feedback, Formal Analysis
Keith Cheveralls
Critical Feedback, Validation
Megan L. Hochstrasser
Editing
Evan Kiefl
Conceptualization, Formal Analysis, Investigation, Methodology, Software, Visualization, Writing
