🔬 The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub
TL;DR
Alex Rives demonstrates how the 'bitter lesson' of AI scaling applies to protein biology, showing that massive transformer models trained on billions of evolutionary sequences develop emergent world models capable of predicting structure, function, and designing novel antibodies without prior biological knowledge.
📈 Scaling Laws & The Bitter Lesson 3 insights
Scaling drives capability emergence in protein models
Training transformers on billions of protein sequences follows the same empirical scaling laws as language models, with new biological prediction capabilities emerging as model size increases by orders of magnitude.
Evolution provides the training signal
The model learns by predicting masked amino acids across evolutionary sequences, capturing the constraints that govern protein structure and function without explicit biological priors.
Proteins follow distributional patterns
Like words in language, amino acids appear in context sets determined by underlying biological constraints, allowing statistical pattern recognition to reveal hidden variables of protein biology.
🧬 ESMC Cambrian Architecture 3 insights
6B parameter world model open-sourced
ESMC represents a fourth-generation protein language model released under MIT license, trained on 6.8 billion sequences and predicting structures for 1.1 billion representative protein clusters.
Inverse design enables protein engineering
The model functions as a searchable world model that can be inverted to design novel proteins including antibodies and single-chain variable fragments (scFVs) meeting specific functional criteria.
Comprehensive coverage of protein space
The database clusters sequences at 70% identity to resolve structures covering the full diversity of known proteins, adding hundreds of millions of new structural predictions to scientific knowledge.
🔍 Emergent Biological Understanding 3 insights
Hierarchical features mirror biological knowledge
Sparse autoencoder analysis reveals the model developed a hierarchical feature space corresponding to biochemical properties, structural motifs, and functional themes discovered through centuries of biological research.
Cross-evolutionary pattern recognition
The model identifies functional relationships like the nucleophilic elbow across evolutionarily distant protein families, using shared latent variables to represent convergent biological solutions.
Compression drives biological insight
To solve the complex task of predicting masked amino acids across diverse contexts, the model develops hidden variables representing fundamental constraints like structural contacts and functional requirements.
Bottom Line
Train massive transformer models on evolutionary protein sequence data at scale to unlock emergent world models capable of predicting structure, function, and designing novel therapeutic proteins without requiring handcrafted biological priors.
More from Latent Space
View all
🔬 "The Most Innovative Diffusion Research Is Happening in Drug Discovery, Not Image Generation"
Evan Fineberg and Sergey Udov of Genesis Molecular AI discuss how diffusion models have pivoted from image generation to drive breakthroughs in 3D protein structure prediction. They detail how their Pearl model applies LLM-style scaling strategies—including synthetic physics-based training data and inference-time 'thinking'—to solve the historically intractable challenge of predicting how small molecules bind to proteins.
Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen
OpenAI Chief Research Officer Mark Chen discusses the company's research philosophy while cooking Korean tofu stew, emphasizing that scaling laws remain robust, reinforcement learning excels in objective domains, and successful research organizations balance top-down vision with bottom-up conviction.
The Agent Cloud: Databricks’ Bet on the Future of AI — Matei Zaharia and Reynold Xin
Matei Zaharia and Reynold Xin detail Databricks' open-source 'Agent Cloud' platform (Omnigen), arguing that standardized protocols and persistent infrastructure—not just better models—will determine which enterprises successfully deploy collaborative, secure AI agents at scale.
AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan
Gray Swan co-founders Zico Kolter and Matt Fredrikson explain why AI systems require a fundamentally different security approach than traditional software, highlighting how their automated red teaming system 'Shade' has begun to outperform human experts at finding model vulnerabilities. They emphasize the urgent need to treat AI agents as inherently untrusted entities capable of correlated failures across the software ecosystem.