Skip to main content
Back

LeetCUDA Hits 11.5k Stars Teaching GPU Kernel Writing the Hard Way

LeetCUDA is an open-source CUDA learning repo with 200+ kernels, PyTorch bindings, and curated GPU programming blogs that has grown to 11.5k stars

Repo, Full Access
  • What it is: LeetCUDA is an open-source CUDA learning repo with 200+ kernels, PyTorch bindings, and 100+ GPU programming blogs, now at 11.5k stars.
  • Difficulty ladder: Kernels are graded Easy to Hard++ covering element-wise ops, GEMM, and full FlashAttention-2 using raw MMA PTX assembly.
  • HGEMM performance: The included half-precision matrix multiply reaches 98–100% of cuBLAS TFLOPS on RTX 4090, L20, and RTX 3080 Laptop.
  • FlashAttention variants: Six progressively optimized FA-2 implementations reduce on-chip SRAM from 4×BrD down to O(Br×16), with the best variant spun out as ffpa-attn.
  • Interview mode: A dedicated kernels/interview/notes-v2.cu compiles to a self-contained verification harness covering 25+ kernels on Ada and Hopper GPUs.
  • Free and open: GPL-3.0 licensed, community contributions welcome; each kernel ships with a Python test and PyTorch binding.

LeetCUDA is an open-source CUDA learning resource built by DefTruth (xlite-dev) that has quietly become one of the most-starred GPU programming repositories on GitHub, crossing 11.5k stars and 1.2k forks. The premise is simple but rare: instead of a slide deck or a textbook, it gives you runnable, well-commented CUDA kernels that go from trivial element-wise ops all the way to hand-rolled FlashAttention-2 using raw MMA PTX assembly, each wired up to PyTorch via Python bindings so you can benchmark them immediately.

The gap it fills

The deep-learning stack is deeper than ever: Python on top, C++/CUDA at the bottom, and a large gap in between where most engineers get lost. LeetCUDA is a self-contained repo that tries to shrink that gap. It is not another "CUDA-101" slideshow , it is a living notebook that couples 200+ hand-written kernels with PyTorch integration, end-to-end benchmarks, and a curated blog roll that explains why every line of code looks the way it does.

The name is intentional. It borrows the LeetCode framing , graded difficulty, one problem at a time , and applies it to GPU kernel writing. The kernels guide you through a step-by-step progression, ranging from easy to very challenging topics, with the workflow for each topic being: custom CUDA kernel implementation, PyTorch Python bindings, then run tests.

What's inside

The difficulty ladder spans five tiers:

  • Easy / Medium: Element-wise ops, matrix transpose, warp/block reduce, NMS, ReLU, GELU, Swish, LayerNorm, RMSNorm, online softmax, dot product, and embedding , covering FP32, FP16, BF16, and FP8.