AI Security After Codex and Claude Code — Zico Kolter & Matt Fredrikson, Gray Swan

Treating AI as Inherently Untrusted Entities

Unlike traditional cybersecurity, Gray Swan approaches AI models as software with unique behavioral vulnerabilities that can be tricked like humans, introducing novel risks when integrated into networks and granted autonomous tool access.

Systemic Risk from Model Concentration

The widespread deployment of specific agents like Codex and Claude Code creates dangerous correlated failure risks, where a single exploit can simultaneously compromise numerous systems unlike distributed traditional software flaws.

Beyond Traditional Cybersecurity

AI security focuses on the model's own susceptibility to jailbreaks and prompt injection rather than using AI merely as a tool to identify bugs in existing codebases.

Automated Systems Surpass Human Capabilities

Gray Swan's specialized system 'Shade' now finds significantly more model breaks than human red teamers in fixed-time competitions, marking a shift where trained AI exceeds human adversarial capabilities.

Why Frontier Models Make Poor Red Teamers

Major AI models refuse adversarial tasks due to safety training, making them ineffective at red teaming compared to explicitly trained specialized models designed to bypass normal behaviors.

Crowdsourced Adversarial Testing

The company operates a 15,000-member 'Arena' community hosting prize challenges to generate training data and identify vulnerabilities for frontier labs like Anthropic.

Indirect Prompt Injection in Coding Agents

Testing revealed that coding agents remain vulnerable to indirect prompt injection when fetching untrusted web content, allowing attackers to hijack objectives, leak data, or steal credentials.

Alien Intelligence Failure Modes

AI systems exhibit fundamentally different intelligence than humans, failing in ways people never would and vice versa, requiring new security frameworks incapable of being predicted by human intuition alone.