Inference, Diffusion, World Models, and More | YC Paper Club

Inference speed determines peak intelligence

As AI systems scale via test-time compute and reasoning (like RL), tokens-per-second becomes the hard ceiling on deliverable intelligence, not merely an operational cost.

RL compute exceeds pre-training

Reinforcement learning already surpasses pre-training compute requirements, and since RL fundamentally wraps inference, generation efficiency is now the critical path for advancement.

Parallelizing sequential dependencies

SSD eliminates the bottleneck where drafting must wait for verification by having the small draft model predict likely verification outcomes and begin drafting the next round while the large model verifies the current one.

Predicting the verifier's behavior

The algorithm predicts which tokens the large model will accept—including the bonus token—with 80-90% accuracy by analyzing the draft model's token distributions, allowing complete latency hiding.

Intelligent cache miss handling

Rather than naively falling back to standard speculation on mispredictions, SSD optimizes compute allocation across plausible prefix lengths to maximize hit rates and maintain speed.

300 tokens per second on large models

The SSD implementation achieves approximately 300 tokens per second on Llama 3 70B using only four H100s, significantly outperforming existing open-source inference engines like SGLang.

Throughput and latency improvements

Unlike vanilla speculative decoding which primarily reduces latency, SSD improves both latency and throughput simultaneously by fully overlapping draft and target model computation.