CUDA Training — 3 Days, 11 Parts, from vector_add to Flash Attention
Three days, nine CUDA/Triton kernels, and more than 70 benchmark runs on T4 and L4 GPUs. This is an 11-part series that reconstructs the whole learning path from start to finish.
Results by the Numbers
- 9 — CUDA / Triton kernels
- 4.79× — Flash Attention speedup (N=4096, T4)
- 65× — HBM traffic reduction
- 54 TFLOPS — Triton fp16 matmul
- 0.86× — compared with cuDNN FA-2 (L4, 300-line Triton)
Structure
PART I · Memory — vector_add, pageable vs pinned PART II · Parallelism — reduction, matmul PART III · Fusion & Attention — softmax fusion, Flash Attention capstone PART IV · Integration — PyTorch custom op, Triton port, MHA causal FA PART V · Essays — Triton vs CUDA, the secret behind 300 lines
To preserve the custom design and typesetting, I grouped the material into a separate portal instead of expanding the whole series inside this blog post.