Chris Fregly's Open-Source GPU Book Helps Engineers Fix 30% Goodput
A hands-on repo pairs an O'Reilly book with 20 chapters of GPU kernel tricks, PyTorch tuning, and trillion-parameter inference scaling recipes.
Takeaways
- Companion repo to O'Reilly book AI Systems Performance Engineering, Apache 2.0 licensed at 1.6k stars.
- 20 chapters spanning CUDA kernels, PyTorch tuning, distributed training, and multi-node inference scaling.
- Ships a 200-plus item performance checklist for reproducing wins and preventing regressions.
- Covers vLLM, SGLang, TensorRT-LLM, and NVIDIA Dynamo with disaggregated prefill/decode and paged KV cache.
- Includes FlashMLA, ThunderMLA, and FlexDecoding kernel walkthroughs plus Triton and CUTLASS examples.
- Backed by an active monthly meetup with recorded sessions and current-hardware slides.
If you have ever stared at an Nsight profile wondering why your H100s are only hitting 30% goodput, there is a new open-source resource aimed squarely at that pain. cfregly/ai-performance-engineering is the companion repo to a new O'Reilly book by Chris Fregly, collecting thousands of lines of PyTorch and CUDA C++ examples alongside a 200-plus item performance checklist that covers the full GPU stack.
The project has crossed 1.6k stars, ships under Apache 2.0, and breaks down to roughly 83% Python and 10% CUDA. Twenty chapters walk from hardware fundamentals up through multi-node inference, with meetup slides and recorded sessions layered on top.
Chasing goodput, not FLOPs
The argument running through the material is that modern AI systems demand more than raw FLOPs and require goodput-driven, profile-first engineering across hardware, software, and algorithms. Goodput here means the useful throughput you actually get after stalls, communication overhead, and memory bottlenecks, rather than the theoretical peak on a spec sheet.
Concretely, the book teaches you to diagnose real bottlenecks with Nsight and PyTorch profilers, squeeze bandwidth and memory, and use compiler stacks like PyTorch plus OpenAI Triton to craft high-impact kernels. On the serving side, it walks through high-throughput inference with vLLM, SGLang, TensorRT-LLM, and NVIDIA Dynamo, including disaggregated prefill/decode and paged KV cache.