MLOps 101: Platforms and Processes for Building AI | NVIDIA GTC

Define success metrics upfront

Establish clear evaluation criteria before modeling to avoid confirmation bias and subjective assessments of model performance.

Iterative cyclical development

Production feedback loops require continuously revisiting prior stages like data collection, retraining, or model architecture selection.

Data preparation is critical

Significant time must be spent federating, cleaning, and labeling data to create a validated 'golden dataset' before any model training begins.

Avoid 'right for wrong reasons'

Models can appear effective while actually relying on spurious correlations, similar to how trial by ordeal worked through self-selection bias rather than divine intervention.

Prevent target leakage

Training on information unavailable during prediction time creates misleadingly high accuracy, analogous to fortune tellers using leading questions to gather hidden information.

Beware default parameter superstition

Blindly using library defaults or outdated prompt templates without understanding underlying mechanics leads to suboptimal or broken results as software evolves.

Respect interface contracts

Component-level correctness fails when system-level assumptions about data ranges, units, or extreme values are violated, as seen in the Ariane 5 and Mars Climate Orbiter disasters.

Monitor for feedback loops

Algorithmic systems can create destructive resonance where user behavior amplifies system outputs, similar to how the Millennium Bridge collapsed from synchronized pedestrian movement.

Implement proper DevOps practices

The Knight Capital $440M loss demonstrates that AI systems require rigorous version control, staged rollouts, and complete deployment coverage to prevent catastrophic failures.

Teleological vs mechanistic explanations

Teleological evaluation asks if the system works, while mechanistic analysis investigates how it works, with the latter essential for debugging unexpected feature reliance.

Identify unexpected feature reliance

Models often exploit background cues rather than intended objects, such as classifying dress shoes as running shoes based on the presence of a track rather than the shoe itself.