Stanford AA228V I Validation of Safety Critical Systems I Explainability

AI-squared dominance on large systems

Top leaderboard performers achieved tightly clustered scores (0.70-0.72) using AI-squared verification techniques for large-scale systems, significantly outperforming other approaches.

Advanced geometric methods for small systems

Winning solutions employed zonotopes and PCA-aligned rectangles rather than simple axis-aligned box approximations, capturing more accurate reachable sets.

Second-order Taylor expansions improve accuracy

For medium systems, utilizing Hessian matrices for second-order Taylor expansion provided measurable performance gains over first-order linearization methods.

Post-incident stakeholder pressure

Chief engineers at companies like Waymo or aviation firms face intense scrutiny following rare catastrophic failures after thousands of successful operating hours, requiring immediate explanations to CEOs, investors, and regulators.

Three critical post-failure questions

Engineers must definitively answer why the specific failure occurred, what system or dataset modifications will prevent recurrence, and how to formally guarantee to stakeholders that the issue is resolved.

Limitations of leave-one-out analysis

Simple ablation studies that zero out individual noise variables at specific time steps often fail to identify failure causes because catastrophic outcomes frequently stem from correlated patterns across multiple consecutive steps.

Group-based noise attribution required

Analyzing groups of time steps rather than isolated events is necessary to detect redundancy and synergy effects in noise sequences that drive systems into failure regimes.

Game theory foundations for ML explainability

Shapley values from 1950s cooperative game theory provide a mathematically rigorous framework to attribute system failures to specific input features by averaging performance across all possible subsets of variables.

Handling redundancy and synergy

Unlike simple ablation, Shapley values correctly account for scenarios where multiple noise variables are redundant or exhibit synergy, providing precise numerical attribution for each variable's contribution to failure.

Computational challenges in long trajectories

Applying Shapley values to safety-critical trajectories with 40+ time steps presents significant computational challenges due to the combinatorial explosion of subset evaluations required by the method.