Cooking with OpenAI’s Research Chief: AGI, o1, Evals, and Scaling Laws — Mark Chen

Replicate papers to build research taste

Chen advises aspiring researchers to replicate papers they admire down to exact training curves and loss metrics, as this reveals hidden techniques and develops intuition better than formal credentials.

Trading skills transfer to research

His quantitative trading background provided valuable experience in "brutal hard optimization" and attention to detail in environments where you cannot cheat the metrics.

PhDs are optional

OpenAI prioritizes creative problem-solving over formal credentials, actively hiring researchers without traditional ML backgrounds and training them internally.

RL thrives on objective truth

Reinforcement learning excels in domains with clear right/wrong answers like math and coding but struggles in subjective fields like creative writing where evaluation criteria vary between experts.

Moving beyond human benchmarks

As models saturate standardized tests like the IMO, evaluation shifts toward real-world research capabilities, novel theorem discovery, and generating cross-domain insights that exceed top human experts.

Coding tests real-world capability

Chen identifies software engineering as the critical domain for testing long-horizon, high-context learning in practical settings.

Scaling laws remain valid

Chen strongly rejects "pre-training is dead" narratives, noting that scaling laws have persisted across nearly ten orders of magnitude and will continue through improved engineering and data techniques.

Betting on reasoning required conviction

Developing the o1 model faced internal resistance because the existing pre-training plus post-training paradigm was already effective, requiring significant vision to pivot toward reasoning architectures.

Stable roadmap, flexible execution

OpenAI maintains consistent high-level research pillars (pre-training, RL, alignment) but reassesses implementation details and compute allocation every one to two months.

Meritocratic leadership

Research managers are selected from the organization's best researchers, enabling effective top-down steering through earned credibility rather than hierarchical authority.

Embrace being wrong

The culture encourages bottom-up innovation where any researcher can redirect the roadmap by presenting compelling evidence that challenges leadership assumptions.

Directive compute allocation

The company focuses resources on three to five major bets per organization while preserving flexible compute pools for emergent ideas from individual researchers.