Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith

From developer side project to sustainable enterprise

Started in 2023 while building a legal AI assistant to solve their own need for quality-cost-speed tradeoff data, initially running on personal funds with hundreds of dollars in compute costs before growing to 20+ employees funded by enterprise subscriptions and private benchmarking services.

Strict independence through dual revenue streams

No lab pays to appear on the public website; revenue comes instead from enterprise advisory subscriptions (covering deployment decisions like serverless vs. self-hosted) and private benchmarking for AI companies testing their own models.

The 'mystery shopper' integrity protocol

To prevent labs from manipulating results when providing private API endpoints, they register anonymous accounts on non-company domains to verify that public endpoints perform identically to preview versions, ensuring no special treatment.

Controlling for benchmark gaming

Discovered labs used inconsistent prompting methodologies—such as Google using 32 unpublished chain-of-thought examples for Gemini Ultra to beat GPT-4 on MMLU—necessitating in-house standardized evaluation across all models.

Statistical rigor drives exponential costs

Moved beyond single-run evaluations to multiple repetitions achieving 95% confidence intervals (±1 point precision), contributing to non-linear cost increases as models and evaluation complexity grew beyond the 'hundreds of dollars' initial budget.

Addressing evaluation bias systematically

Implement sophisticated answer extraction to parse model outputs without penalizing reasoning for formatting errors, randomize multiple-choice answer order to eliminate position bias, and control for temperature variance that can create enormous score swings on small question sets.

End of the OpenAI monopoly

Market structure shifted from OpenAI's year-long untouchable dominance to intense multi-polar competition, with DeepSeek's V3 release (Boxing Day 2024) proving non-US labs could reach frontier capabilities, followed weeks later by R1's reasoning breakthrough.

Benchmark saturation cycles

Intelligence Index evolved from V1 through V3 because tasks like HumanEval became trivial for modern models; current focus shifting to agentic capabilities, long-context reasoning, hallucination detection, and economically valuable use cases rather than academic Q&A.

Quality-per-dollar revolution

Progress in model efficiency means even small models today solve 100% of problems that required frontier models two years ago, driving down intelligence costs across all tiers while making raw capability scores less meaningful than task-specific performance.