LESSON 06 · 2026.04.18 · T4 · CAPSTONE

Flash Attention — five lessons converging into 80 lines

Tiled matmul + online softmax + HBM-traffic reduction. Everything lessons 1–5 built, compressed into a single kernel. 4.79× faster and 65× less memory traffic at N=4096.

Two implementations

• naive — 3 kernels (QK → softmax → PV). Fully materializes intermediates S and P (N×N) in HBM.
• flash — single kernel. Q in Br=64 row blocks, K/V in Bc=32 column blocks. Each tile accumulates Q@K^T, online softmax, and P@V in one pass. S and P never touch HBM.

Results

GFLOP/s (by compute count): naive 169 → 366, flash 115 → 1754 (22% of T4 peak). Accuracy: both max_abs_err < 5e-7.

Lesson 1 · HBM reduction scales N² → N·d

At N=4096, d=64:

• naive HBM ≈ 4N² + 4N·d = 260 MB
• flash HBM ≈ 4N·d = 4 MB

Ratio 65×, widening quadratically in N. This is the whole of FA. The rest is engineering that converts this reduction into time.

Lesson 2 · Crossover — naive is faster at N=512

At N=512, flash is actually slower (0.68×). Reasons:

• naive's S (1 MB) fits in T4's L2 (4 MB) → actual HBM loads are far below theoretical
• flash rotates K/V for each Q block → recompute overhead
• Br=64 means too few blocks to fill T4's 40 SMs → low occupancy

"FA is always faster" is false. It wins once sequence length is long. That's why FA is decisive for real LLM prefill (N=4096–32k).

Lesson 3 · HBM reduction vs wall-clock speed

At N=4096, HBM drops 65× but wall-clock only 4.79×. Why?

• naive isn't really HBM-bound (effective 22 GB/s → 7% of T4). L2 absorbs most of it.
• FLOPs are identical — flash only cuts data movement, not compute
• flash plateaus at 1.75 TFLOPS → entering a compute-bound regime

FA's big wins appear on high-bandwidth + high-peak silicon like H100. On T4 the savings are real but not dramatic.

Lesson 4 · Five lessons converge in one kernel

Lesson 01 vector_add  → coalesced load pattern (FA's Q/K/V loads)
Lesson 02 memory      → HBM↔L2 traffic awareness (the reason FA exists)
Lesson 03 reduction   → warp reduce (row max/sum)
Lesson 04 matmul      → tiled matmul (S = Q@K^T, O += P@V)
Lesson 05 softmax     → online (max, sum) update (heart)
Lesson 06 flash       → fusion of the five above

All of that sits inside one 80-line kernel. That's why the FA paper is often described as "a sharp combination of not-particularly-novel techniques."

LLM-serving translation

• Prefill: N in the thousands to tens of thousands. Without FA the memory itself runs out. This is where vLLM / TensorRT-LLM call FA-2/FA-3.
• Decode: seq_len = 1 (the current query). Q is so small that FA's tile structure is overkill. Other optimizations like Paged Attention matter more.

Our single-kernel FA explains the dynamics on the prefill side. This closes "CUDA Level 1." Next lesson onward we attach this to PyTorch.