LESSON 04 · 2026.04.18 · T4

Matmul — from memory-bound to Tensor Cores

0.4 → 7.9 TFLOPS. Same GPU, same matrix. Across four implementations the ceiling shifted twice. The spot where the roofline axis jumps vertically.

GPU · T4 peak · FP32 8.1 · TC 65 TFLOPS sweep · 4 sizes × 4 versions

Four versions

v1 naive — 1 thread / 1 output. global memory only
v2 tiled — 32×32 shared-memory tile (AI = 8 FLOP/byte)
v3 register — block 128×128, thread tile 8×8, BK=8 (AI = 32 FLOP/byte)
v4 tensor — WMMA API, block 64×64, FP16 inputs → FP32 accumulate

TFLOPS matrix

N	v1 naive	v2 tiled	v3 reg	v4 tensor
256	0.22	0.56	0.20 ⚠	1.51
512	0.47	0.75	0.95	3.71
1024	0.40	0.64	1.43	3.59
2048	0.40	0.83	2.05	7.93

v3 @ 2048 = 25% of FP32 peak. v4 @ 2048 = 12% of TC peak. The point: plain WMMA alone lands 7.93 TFLOPS.

Lesson 1 · Occupancy trap — v3 is slower than v2 at N=256

v3's block tile is 128×128. At N=256 that's just 4 blocks. 36 of T4's 40 SMs sit idle. v2's 32×32 tiles → 64 blocks → all SMs working.

Bigger tiles mean higher AI — good. But you must keep blocks ≥ 2 × SM count. Miss that and you go slower instead.

LLM-serving implication: big matmuls (FFN) are fine, but the small-batch decode attention matmul needs a "small-tile variant."

Lesson 2 · What v3 is really doing

Each thread holds an 8×8 output tile in registers. That gives:

8× fewer reads of the same shared-memory value compared to v2
Threads per block: 1024 → 256 (4× fewer) → more resident blocks per SM = occupancy restored
Block tile: 32×32 → 128×128 (16×) → AI up 4×

These three compound to 2.05 TFLOPS at N=2048 — close to cuBLAS SGEMM's 3–4 TFLOPS.

Lesson 3 · v4 is a new ceiling — Tensor Cores

One mma_sync instruction = 16×16×16 matrix product = 4096 FMAs, done in ~8 cycles. Per warp-cycle, that's 16× the throughput of FP32 FMAs.

Why we only hit 12% of peak:

Fragment loads are not in swizzled layout
No double buffering (next tile HBM→shared overlapped with current tile mma)
Block tile 64×64 is small (AI limited)

CUTLASS implements all three. Months of work.

Lesson 4 · The cost of precision

version	max_abs_err @ N=1024
v1, v2, v3 (FP32)	7.6e-5
v4 (FP16 input)	1.4e-2 · 180×

FP16 inputs → 10-bit mantissa → about 3 decimal digits of precision. Fine for LLM inference, training needs BF16/FP32 mix. This is why AWQ, GPTQ, and FP8 quantization exist. Lower precision → faster TC → if output is nearly the same, net win.

Roofline intuition

                    │       Tensor Core peak ━━━━━━━━━━
perf ▲              │
(TFLOPS)            │                                  v4 ●
                    │                                (7.9)
                    │      ← FP32 peak 8.1 ──────────────
                    │                         v3 ●
                    │                       (2.0)
                    │                 v2 ●
                    │              (0.8)
                    │      v1 ●
                    │    (0.4)
                    └────────────────────────────────────▶
                       low AI                       high AI

Up through v3 is a bandwidth fight; v4 jumps the axis vertically — a different ceiling. LLM inference runs almost entirely in the bottom two rows. v1–v4 is the ladder up to there.

Prev · 03reduction Next · 05softmax & fusion