Playground in Prod - Optimising Agents in Production Environments — Samuel Colvin, Pydantic

Pareto Frontier Breeding Strategy

Jepper optimizes prompts using genetic algorithms that selectively breed candidates from the Pareto frontier of best performers, similar to selective racehorse breeding rather than random mutation.

String Optimization Beyond Text

The library optimizes any string value, whether simple text prompts or JSON data containing complex structured configurations.

Structured Configuration Management

Logfire's managed variables support any object definable by a Pydantic model, extending beyond simple text prompts to enable management of complex structured agent parameters.

AI Observability as Feature Not Category

Colvin argues that AI observability will eventually be absorbed by general observability platforms or AI frameworks, serving as a feature rather than a standalone category.

Autonomous Optimization Pipeline

The platform is evolving toward autonomous agent optimization where variables are tuned directly from the observability interface without manual intervention.

Golden Dataset Over LLM Judges

Deterministic evaluators comparing outputs against verified golden datasets provide more reliable benchmarks than LLM-as-judge approaches, which Colvin describes as 'lunatics running the asylum'.

Political Dynasty Extraction Demo

The demonstration uses Pydantic AI with structured outputs to analyze Wikipedia data for UK MPs, specifically optimizing prompts to identify ancestral political relationships while filtering out spouses and siblings.

Pydantic Gateway for Model Access

The gateway service provides unified API access to multiple model providers with built-in observability, caching, and fallback capabilities for production environments.