Insights from NVIDIA Research | NVIDIA GTC

Dual-sided organization structure

NVIDIA Research operates 500 people across 'supply side' (GPU technology from circuits to programming) and 'demand side' (AI, robotics, quantum applications driving GPU adoption).

AI hardware genesis

Collaboration with Andrew Ng and Bryan Catanzaro ported deep learning from 16,000 CPUs to 12 GPUs, creating cuDNN and establishing NVIDIA's AI leadership.

Networking pivot against executive resistance

DOE-funded research created NVLink and NVSwitch after Jensen initially rejected networking investments, with technology migrating from research to Pascal and Volta GPUs.

RTX cores moonshot origin

The 'tree traversal unit' research project achieved 100x ray tracing speedup through specialized hardware, rebranded as RTX cores for real-time graphics.

Latency versus throughput spectrum

Batch processing prioritizes tokens-per-dollar (left side) while real-time agentic AI requires interactivity (right side) demanding 100 to 10,000+ tokens per second per user.

Communication dominates latency

For real-time inference, 89% of time is spent on communication versus only 11% on compute and memory, with 500 communication stages per token across 80 layers.

Memory bandwidth constraints

Decode phase inference requires reading every model weight for each single token, creating a memory bandwidth bottleneck that limits throughput.

SRAM-compute fusion

Placing arithmetic units directly at SRAM edges eliminates data movement by performing dot products immediately upon weight retrieval across tiled processing elements.

Static scheduling for speed

Eliminating queuing, arbitration, and routing decisions enables 50 nanosecond on-chip communication latency by advancing activations over wires in pre-determined paths.

Low-latency off-chip links

Reducing bandwidth from 400 to 200 Gbps removes complex DSP and forward error correction, achieving ~100 nanosecond switch traversal versus current multi-microsecond delays.

3D DRAM stacking

Placing DRAM directly atop GPU dies with localized storage above each processing element eliminates data movement energy, targeting 10x reduction in joules per token.

10,000+ token velocity goal

Current systems achieve ~350 tokens/sec while the research prototype targets 16,000 tokens/sec per user to enable real-time reasoning and tree-of-thought AI agents.

Spatial KV cache distribution

Pipelining architecture keeps portions of the KV cache localized to specific chips, minimizing energy-intensive off-chip data movement for batch workloads.