🔬 The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub

Scaling drives capability emergence in protein models

Training transformers on billions of protein sequences follows the same empirical scaling laws as language models, with new biological prediction capabilities emerging as model size increases by orders of magnitude.

Evolution provides the training signal

The model learns by predicting masked amino acids across evolutionary sequences, capturing the constraints that govern protein structure and function without explicit biological priors.

Proteins follow distributional patterns

Like words in language, amino acids appear in context sets determined by underlying biological constraints, allowing statistical pattern recognition to reveal hidden variables of protein biology.

6B parameter world model open-sourced

ESMC represents a fourth-generation protein language model released under MIT license, trained on 6.8 billion sequences and predicting structures for 1.1 billion representative protein clusters.

Inverse design enables protein engineering

The model functions as a searchable world model that can be inverted to design novel proteins including antibodies and single-chain variable fragments (scFVs) meeting specific functional criteria.

Comprehensive coverage of protein space

The database clusters sequences at 70% identity to resolve structures covering the full diversity of known proteins, adding hundreds of millions of new structural predictions to scientific knowledge.

Hierarchical features mirror biological knowledge

Sparse autoencoder analysis reveals the model developed a hierarchical feature space corresponding to biochemical properties, structural motifs, and functional themes discovered through centuries of biological research.

Cross-evolutionary pattern recognition

The model identifies functional relationships like the nucleophilic elbow across evolutionarily distant protein families, using shared latent variables to represent convergent biological solutions.

Compression drives biological insight

To solve the complex task of predicting masked amino acids across diverse contexts, the model develops hidden variables representing fundamental constraints like structural contacts and functional requirements.