Don't Fight Backprop: Goodfire's Vision for Intentional Design, w/ Dan Balsam & Tom McGrath

Don't Fight Backprop

Goodfire advocates shaping loss landscapes so models naturally learn desired behaviors rather than imposing constraints that gradient descent will inevitably circumvent.

Frozen Probe Hallucination Reduction

Their proof-of-concept reduces hallucinations by running detection probes on a frozen copy of the model during training, making it easier to learn correct behaviors than to evade detection.

Safety-First Deployment Stance

While intentional design offers promise, the researchers acknowledge these techniques remain immature and should not yet be applied to frontier models.

Beyond Discrete Features

The field is shifting from sparse autoencoders that label discrete concepts toward mapping continuous geometric manifolds that represent conceptual relationships in latent space.

Manifolds Enable True Generalization

Understanding these geometric structures is necessary for circuit explanations that generalize across all possible inputs rather than merely tracing individual execution paths.

Structure in Representations

Concepts like days of the week form structured geometric patterns such as circular manifolds rather than random disconnected points, driven by co-occurrence statistics in training data.

Separating Memory from Reasoning

Goodfire demonstrated that removing weights specialized for fact memorization can actually improve model performance on certain reasoning tasks.

Debugging Medical AI

Their Prima collaboration revealed that an Alzheimer's diagnosis model relied on DNA fragment length rather than intended biological markers, showcasing interpretability's value for detecting spurious correlations.