Inference Office Hours with SGLang: Performance Optimizations for LLM Serving

NVLink Backbone Architecture

GB200 NV72 utilizes pure NVLink connectivity across 72 GPUs instead of InfiniBand, drastically reducing communication latency for distributed expert parallelism.

Quantization Performance Gains

Transitioning from BF16 to FP8 yields 1.8x speedup, while moving FP8 to NVFP4 delivers an additional 1.9x acceleration with negligible accuracy loss on GBQ and MASS 500 benchmarks.

Record Throughput Metrics

On GB200, SGLang achieves 26,000 input and 13,000 output tokens per second per GPU using 48 decode ranks, significantly surpassing previous generation baselines.

PD Disaggregation Strategy

Separating prefill and decode into distinct node types prevents task preemption, enables balanced data parallel attention with VSS, and utilizes RDMA for efficient KV cache transfer.

Two-Batch Overlap Technique

Splitting batches into microbatches enables simultaneous computation and communication across dual CUDA streams, hiding latency by overlapping attention operations with data transfers.

Optimized Kernel Selection

Prefill stages leverage Flash Infer Cutlass for FP8/NVFP4 gemms while decode stages utilize Deep Gemm and TensorRT-LLM attention kernels specifically optimized for Blackwell architecture.

Zero-Overhead Speculative Decoding

Spec V2 eliminates GPU idle bubbles by running the overlap scheduler concurrently with speculative decoding via EagleWorkerV2, delivering 20% end-to-end speed improvements across all workloads.

Piecewise CUDA Graphs

Prefill operations now exclude attention kernels from CUDA graphs to avoid variable sequence padding, capturing only non-attention components while overlapping attention launch overhead with CPU work.

Unified Feature Compatibility

SGLang is refactoring memory pools and parallelism modules to ensure PD disaggregation, speculative decoding, and CUDA graphs can operate simultaneously using default configuration arguments.