Notion’s Sarah Sachs & Simon Last on Custom Agents, Evals, and the Future of Work

Four complete rebuilds since 2022

Notion attempted to build AI agents immediately after gaining GPT-4 access in late 2022, but abandoned early attempts involving fine-tuned function calling because models lacked native tool support and sufficient context windows.

The Sonnet 3.6/3.7 unlock

The project only became production-viable after Anthropic's Sonnet 3.6 and 3.7 emerged early last year, finally providing the reliability necessary for background-running agents that operate without human supervision.

Learning to swim with the current

The team developed crucial intuition for distinguishing between 'swimming upstream' against model limitations versus preparing infrastructure early for anticipated capability improvements.

Radical low-ego codebase

Sachs emphasizes a culture where engineers happily delete their own code and avoid writing design docs for political advancement, enabling the team to abandon approaches immediately when model capabilities shift.

The Simon vortex methodology

Last operates a high-velocity prototyping 'vortex' where direction changes daily, staffed by senior engineers who rotate between experimental work and productionization without strict management boundaries.

Decentralized ideation over hierarchy

Leadership focuses on clarifying objectives rather than gatekeeping ideas, allowing engineers who identify user pain points to prototype solutions without seeking hierarchical approval from product leadership.

MCP for narrow permissioned tools

They view Model Context Protocol as ideal for lightweight agents requiring strict permissions, contrasting with full coding agents that need compute runtimes and sandboxed environments.

P99 transcript analysis ritual

Every Friday, the team examines the most token-intensive failed agent transcripts to identify specific failure modes and ruthlessly cut tasks that exceed current model capabilities.

Solving enterprise permission intersections

Building for enterprise required solving complex access control where agents shared in Slack channels must respect the intersection between channel memberships and document-level permissions.

The software factory vision

Notion is exploring automated workflows where multiple agents collaboratively develop, debug, merge, and maintain codebases with minimal human intervention, effectively bootstrapping their own capabilities.

Coding agents as the AGI kernel

They view coding agents as the fundamental 'kernel' for AGI, where agents that can write and maintain software effectively create their own tools and extend their own functionality.

The Datadog analogy

Like Datadog provides observability expertise atop AWS infrastructure, Notion aims to be the expert collaboration layer atop raw AI capabilities, focused specifically on how enterprise teams actually work together.