LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize

OpenTelemetry-first instrumentation

Arize leverages OTel standards with auto-instrumentation requiring only one line of code to capture traces and spans across any framework, creating audit records of agent behavior since code alone cannot audit non-deterministic systems.

Multi-layer visibility requirements

Comprehensive observability requires examining granular traces (individual LLM calls), sessions (state and conversation history), and trajectory distributions (all possible agent paths) to understand back-and-forth states and branching logic.

Distributional path analysis

Viewing aggregations of all agent instantiations reveals what percentage of traffic flows down specific branches, helping identify path-dependent latency issues or component ordering errors that single trace views miss.

Five flavors of signal

Effective evaluation combines LLM-as-judge, human feedback, golden datasets for domain-specific quality tuning, deterministic logic checks (like JSON validation), and business metrics covering revenue, cost savings, and time efficiency.

Scoped evaluation depths

Evaluations operate at span level (single component I/O), multi-span (data passing between agents), trajectory level (end-to-end process completion), and session level (conversation satisfaction and state machine behavior).

Role-based collaboration model

Technical users should handle framework coding while domain experts and product managers define evaluation criteria through no-code interfaces, allowing each group to work within their expertise.

Systematic improvement loops

Experimentation involves testing changes to prompts, models, and orchestration configurations against curated datasets to prevent regressions, where fixing one issue often creates unexpected failures in non-deterministic systems.

Code-native workflows

The industry is moving away from dashboards toward CLI tools and coding agents (like Arize's 'Alex') that programmatically analyze traces, detect issues such as high latency or errors, and run evaluations directly in developers' existing environments.

The fully automated flywheel

The ultimate goal is complete automation where AI agents handle observability, generate appropriate evaluations dynamically based on context, and fix systems without human intervention, compressing the entire improvement cycle into an autonomous process.