Reiner Pope of MatX on accelerating AI with transformer-optimized chips

Transformers and talent originated at Google Brain

Nearly every major AI researcher over 30 passed through Google Brain, creating the research foundation for modern LLMs while TPUs provided the necessary parallel compute infrastructure specifically designed for neural nets rather than graphics.

TPU v1 launched in 2016 with remarkable speed

A skeleton team of 20-30 people designed Google's first AI chip in roughly 18 months, creating a minimal viable product that predated the Transformer era but established the mechanical sympathy between parallel hardware and emerging model architectures.

Hardware demands mechanical sympathy from software

Modern chips contain hundreds of billions of transistors requiring hundreds of clock cycles to traverse, making hardware fundamentally parallel and requiring software architectures that maximize parallelization rather than sequential processing.

Founded by elite Google chip architects

Reiner Pope and Mike (Google's former chief chip architect) started MatX to abandon general-purpose constraints and build silicon specifically optimized for large matrix operations and low-precision arithmetic required by modern LLMs.

Hybrid memory solves the latency-throughput paradox

Unlike existing chips that force a choice between high-latency HBM (Google/Amazon/NVIDIA) or low-throughput SRAM (Groq/Cerebras), MatX stores weights in SRAM and inference data in HBM to simultaneously achieve low latency and economic throughput.

$500M Series B targets gigawatt-scale production

Led by Jane Street and Leopold Aschenbrenner's Situational Awareness, the funding transitions the company from $100M small-batch prototyping to supply chains capable of shipping multiple gigawatts of compute annually.

Tokens-per-dollar determines model quality

Throughput economics constrain the quality of AI that can be trained and served within fixed budgets, while latency directly impacts user engagement—Google's internal research shows even 50-millisecond delays measurably reduce usage.

Supply chain crunches threaten AI buildouts

Critical bottlenecks include HBM from the big three vendors (Hynix, Samsung, Micron), logic dies from TSMC, and rack manufacturing involving power delivery, cooling, and high-speed cable infrastructure.

Order-of-magnitude speed improvements are imminent

Current HBM-based chips operate at 10-20 milliseconds per token versus 1 millisecond for SRAM-based systems, with next-generation architectures promising to bridge this gap and deliver 10x faster inference within three to five years.