Stanford Robotics Seminar ENGR319 | Spring 2026 | Towards Trustworthy Autonomy

Mass deployment of learning-based robotics

Waymo now performs nearly 500,000 rides weekly in the US, demonstrating the shift from controlled prototypes to scalable learning-enabled systems operating in messy real-world environments.

Dangerous black-box semantic failure modes

Modern systems fail dangerously by driving into floods, stopping for sunsets mistaken for traffic lights, or braking for billboards displaying stop signs due to spurious correlations in training data.

Semantic anomalies versus physical safety risks

Unlike physical safety involving obstacle avoidance, semantic anomalies arise from holistic context—such as inactive traffic lights on trucks or plastic utensils placed in ovens—causing system-level confusion.

Why traditional out-of-distribution detection fails

Standard out-of-distribution methods fail on semantic anomalies because these scenarios are not visually novel, and models remain highly confident even when context renders their predictions unsafe.

Real-time embedding-based anomaly detection

Small language models like MPNet and BERT extract semantic embeddings to detect anomalies at 40Hz by comparing current observations against databases of prior robot experiences.

LLM chain-of-thought for safety interventions

Large language models provide zero-shot chain-of-thought reasoning to assess the safety-criticality of detected anomalies and select appropriate recovery behaviors from predefined sets.

Integration with Model Predictive Control

The system maintains dynamically feasible recovery sets with overlapping fallback trajectories, ensuring the robot remains safe during the latency period while waiting for LLM reasoning output.

Embedding detectors outperform pure LLM reasoning

On synthetic benchmarks, embedding-based detectors achieve higher accuracy than generative LLM reasoning alone, with performance scaling linearly as more nominal concepts are added to the experience database.

Real-world quadrotor safety validation tests

Hardware experiments demonstrate context-aware decisions, such as diverting to alternate landing zones when other drones occupy target boxes while proceeding past novel objects like keyboards deemed safe by the LLM.