Goodfire AI’s Bet: Interpretability as the Next Frontier of Model Design — Myra Deng & Mark Bissell

Beyond black-box analysis

Goodfire defines interpretability broadly as the "science of deep learning," extending methods beyond post-hoc analysis to cover the entire AI development lifecycle including data curation during training and guiding the learning process itself.

Surgical model editing

The company focuses on enabling precise modifications to model internals—such as removing specific bias vectors (e.g., political slants) or behaviors—rather than retraining entire models or relying on prompting.

Preventing training failures

Interpretability tools can catch unintended post-training behaviors like sycophancy, reward hacking, and "grokking" (memorization vs. generalization), addressing issues like the "40 Glaze" controversy through internal monitoring rather than external observation.

SAEs show surprising limitations

In production testing, probes trained on raw activations sometimes outperformed Sparse Autoencoder (SAE)-based probes for detecting hallucinations and harmful intent, challenging assumptions that unsupervised features always capture concepts more cleanly.

Steering requires robustness

Early steering APIs fell short of black-box techniques like fine-tuning, prompting the team to develop more powerful control mechanisms that can scale to trillion-parameter models like Kimi K2 (requiring 8x H100s for deployment).

Efficiency over guardrails

Internal probes add negligible latency compared to separate guardrail LLM judges, making them viable for real-time production monitoring where external model calls would be too slow or expensive.

Rakuten's PII detection pipeline

Japan's top e-commerce platform uses Goodfire for token-level PII scrubbing in live traffic, handling both English and Japanese queries while using synthetic training data (to avoid privacy violations) with evaluation on real customer data.

Real-world complexity

Production revealed challenges absent in research: Japanese tokenization quirks, synthetic-to-real domain transfer requirements, and the need for token-level (not sentence-level) classification to precisely scrub private information.

Failure-driven research agenda

The team identifies where current ML methods fall short in production, applies state-of-the-art interpretability techniques, and when those fail, uses the gaps to determine fundamental research priorities—such as developing alternatives to SAEs.

From observation to design

The ultimate goal is shifting interpretability from post-training "poking at models" to active training-time guidance, using understanding of internal representations to intentionally design safer, more capable models rather than merely analyzing finished ones.