Stanford Robotics Seminar ENGR319 | Spring 2026 | Ingredientsfor Long-Horizon Robot Autonomy

Robots Master Tasks But Not Jobs

Current systems achieve high dexterity on short operations like unlocking locks, but fail at extended objectives humans would actually delegate, such as doing groceries or assembling server racks.

Three Critical Missing Ingredients

Long-horizon autonomy requires primitive memory to track completed steps, extremely high individual skill success rates to survive statistical chaining over time, and robust generalization to handle novel state combinations.

Entropy Increases With Task Duration

As task horizons extend, the probability of encountering exact training scenarios drops, placing exponentially higher demands on a system's ability to generalize to unforeseen situations.

Naive Memory Approaches Break Systems

Simply feeding historical observations into sequence models creates crippling latency for real-time control and causes distribution shifts, as policies witness their own mistakes rather than perfect human demonstrations.

Dual-Stream Memory Mimics Human Cognition

MEM implements dense, compressed visual tokens for short-term detail recall alongside sparse semantic language representations for long-term task tracking over tens of minutes.

Modified Vision Transformer Compresses Tokens

A specialized ViT architecture uses sparse temporal attention to compress 10 seconds of 50Hz multi-camera data from 512,000 tokens down to standard counts without adding new weights, preserving pretrained initialization.

Endless Loops in Partial Observability

Without memory, robots unpacking groceries repeatedly check empty bags because they cannot remember contents after removing their gripper and losing camera visibility.

Time-Agnostic Behaviors Cause Errors

Memory-less policies wash plates endlessly or burn grilled cheese because they cannot track duration or state changes that lack immediate visual differentiation.