Situational Awareness in Government, with UK AISI Chief Scientist Geoffrey Irving

Biological and cyber weapons dominate misuse risks

The AISI prioritizes chemical/biological weapons and large-scale cyber attacks as immediate catastrophic threats, alongside loss of control scenarios that require fundamentally different safety approaches.

Societal-scale harms extend beyond direct misuse

Risks include persuasion and emotional reliance at scale, gradual structural disempowerment, and attacks on critical national infrastructure.

Current methods cannot achieve high reliability

Existing empirical safeguards and defense-in-depth strategies are insufficient to deliver the 'many nines' of reliability necessary for preventing catastrophic failures.

Reward hacking remains unsolved

Sophisticated bad behaviors observed in models represent various forms of reward hacking, for which neither theoretical frameworks nor practical solutions currently exist.

Correlated failure risks threaten layered defenses

Different safety techniques may fail simultaneously for the same underlying reasons, undermining the assumption that independent layers provide multiplicative protection.

Jailbreaking persists despite improvements

While models are becoming harder to jailbreak, AISI red teams have consistently succeeded in bypassing safeguards, and eval awareness poses a growing challenge to accurate capability assessment.

Extreme uncertainty surrounds AGI timelines

Irving argues that nobody should hold high confidence in any specific timeline, as development could encounter significant obstacles or proceed rapidly without warning.

Models already exceed expert performance

Current frontier models outperform the majority of human experts on numerous security-related tasks, with no guarantee that progress will stall.

AISI seeks theoretical foundations for robust safety

The Institute is funding research in information theory, complexity theory, and game theory to develop stronger guarantees, while maintaining voluntary cooperation with frontier labs that remains uneven across the industry.