Moonlake: Multimodal, Interactive, and Efficient World Models — with Fan-yun Sun and Chris Manning

Action-conditioned prediction separates world models from video generators

Unlike Sora-style video generators that predict pixels, true world models must predict the consequences of specific actions minutes into the future, requiring understanding of 3D physics and object permanence.

Long-term consistency requires semantic abstraction

Maintaining coherent game states or simulations for extended periods requires abstract symbolic representations rather than processing raw pixels, as evidenced by human cognitive processing that filters most visual input.

Symbolic abstraction enables five orders of magnitude efficiency gains

While not rejecting the bitter lesson entirely, Moonlake bets that structured reasoning traces incorporating geometry, physics, and affordances can achieve what pixel-only models require exponentially more data and compute to learn.

Current video data lacks essential action labels

Mining observational videos from the internet fails to capture the actions causing state changes, making it difficult to learn causal relationships without expensive action-conditioned simulation data.

Language serves as a cognitive tool for abstraction

Following philosopher Dan Dennett, Manning argues language provides unique symbolic knowledge representation that enabled human evolutionary advantages in planning and tool-building beyond what vision alone provides.

Philosophical divergence from LeCun's visual-centric JEPA

Moonlake fundamentally disagrees with Yann LeCun's dismissal of symbolic representations, asserting that latent visual abstractions alone cannot achieve the causal reasoning and long-term planning necessary for embodied AI.