LESSON 10 · 2026.04.20 · L4

Nine kernels under the knife — what was hiding behind the numbers

Beyond "fast/slow," reaching "why that number." Turning three claims from lessons 1–9 into numbers with nsys timelines and ncu stall counters.

Through lessons 1–9 I wrote ten CUDA / Triton kernels. I benched each and logged numbers like "v4 is 200× faster than v1" and "ours runs at 78 % of SDPA."

But there was a lot I had moved past without actually knowing why the number was what it was:

• I knew atomic is slow — but how much, and in which HW counter does that slowness show up?
• What path creates the pinned-vs-pageable gap?
• Our Triton FA is 78 % of SDPA — where is the 22 % we lose?

This essay is a record of tearing those kernels apart with nsys (timeline profiler) and ncu (per-kernel metric profiler) — without changing a single line of kernel code. Doing this first paid back more per hour than writing another kernel.

Two tools, two viewpoints

nsys looks at the "critical-path balance between kernel and transfer." ncu looks at "whether warps inside a kernel are idle, computing, or waiting." They don't overlap.

Phase 1 · nsys — pageable is especially slow on D2H

Question: Lesson 04 said pinned memory is faster than pageable. How much, in which direction?

Experiment: Run bin/vector_add --n 16M --iterations 5 twice — once with --pageable, once with --pinned — under nsys, then pull .nsys-rep locally and inspect in GUI + CLI stats.

The surprise: pageable H2D at 4.77 GB/s but pageable D2H at 1.33 GB/s — same PCIe, and yet 3.6× slower. The timeline makes the reason obvious:

• Pageable D2H: a 2-hop path — device → pinned bounce buffer → pageable host. That final memcpy-to-pageable isn't SIMD-friendly and mixes in page faults / cache eviction.
• Pageable H2D: reverse — the driver first copies into a pinned staging buffer, then DMAs to the GPU. "host memcpy → pinned" is close to a seq-read from the OS's view, relatively fast.
• Pinned: 0-hop (DMA direct) in both directions, symmetric — converges to 12–13 GB/s.

L4's PCIe Gen4 x16 effective BW ≈ 26 GB/s. Pinned reaches ~50 % of that. Pageable D2H sits at 5 % — a structural tax.

And the kernel time doesn't move (pageable 0.834 ms, pinned 0.836 ms). Pinning touches only the transfer path, not on-device execution — an obvious fact, now confirmed with numbers.

Phase 2 · ncu — atomic's "slowness" is not occupancy, it's lg_throttle

Question: Lesson 02 reduction v1 (atomicAdd per thread) is hundreds of times slower than v4 (warp shuffle + 1 atomic per block). Fine — but what specific HW counter exposes that slowness?

Experiment: Run bin/reduction --n 4M --version {1,4} each under ncu --set detailed --launch-skip 20 --launch-count 1 -k "regex:reduce_v{1,4}_", then compare stall distribution + SOL metrics.

Three surprises:

1. Occupancy is the same. Both sit at 91–92 %. Intuitively you'd think "v1's warps can't launch because atomic blocks them." In fact the warps do launch — and then sit there waiting. That doesn't register in occupancy.
2. DRAM is empty. v1's DRAM is 0.46 %. This kernel is not memory-bound.
3. But L2 hit is 88.74 % — absurdly high. All threads touch the same 4-byte accumulator, so that cache line gets pinned in L2 and keeps hitting. But lots of hits isn't speed — every SM fighting over one line creates serialization.

That shows up as lg_throttle 31.1 % — "local/global memory throttle," a signal that the LSU (load/store unit) is getting back-pressured on the atomic path. In v4, lg_throttle goes to 0 %, the dominant stall flips to long_scoreboard (normal DRAM load wait), DRAM fills to 88 %, and the kernel takes the healthy shape of a memory-bound kernel.

Phase 3 · ncu-tracing the 20 % gap between ours and SDPA

Question: Lesson 09's 4-D causal FA in Triton hit 78–90 % of F.scaled_dot_product_attention. Where is the 22 %?

Experiment: B=1 H=32 N=2048 d=128 causal fp16 (LLaMA-7B mid-range, where the gap was biggest). Profile each under ncu and compare metrics.

First finding — SDPA's backend wasn't cuDNN. Kernel name:

void flash_fwd_kernel<Flash_fwd_kernel_traits<128, 64, 64, 4, 0, 0, half_t, ...>>(Flash_fwd_params)

That's Tri Dao's Flash Attention 2 CUDA implementation — PyTorch 2.11 ships it and dispatches to it on L4 + fp16 + causal. Not cuDNN. So we're actually comparing Triton FA to a seasoned CUDA implementation of the same algorithm.

Stall distribution:

Four places the 20 % gap lives

1. Register pressure → occupancy halved. Autotune picked BLOCK_M=128, pushing registers to 255 (literally the max, on the verge of spill). Resident warps on the SM get halved. SDPA uses BLOCK_M=64, 184 regs/thread, and keeps 2× the warps alive. Occupancy 8.3 % vs 16.2 %.
2. MMA dependency chain (wait 38.6 %). We consume the output accumulator too close to a tl.dot. num_stages is low, so the consumer waits on the producer MMA. SDPA's wait is only 19 %.
3. Register dependency (short_scoreboard 14.9 % vs SDPA 2.2 %). A follow-on effect of #1 — with the register file stuffed, producer-consumer often reference the same physical register.
4. SDPA is already sitting at a "good" bottleneck. math_pipe_throttle 41.5 % — tensor core saturated. That's a better signal than wait: it means they're in the "you'd need more FLOPs to go faster" regime. We don't reach it.

Three lessons from this session

(a) Occupancy is not throughput

Common thread across Phase 2 and Phase 3: had I only looked at occupancy, I'd have made the wrong diagnosis.

• Phase 2: v1 and v4 have the same 91–92 % occupancy, but 218× wall-cycle difference.
• Phase 3: our 8.3 % occupancy is half of SDPA's 16.2 %, but that's a side effect of "big tile + high register pressure," not "empty warp pool."

Occupancy caps "how many warps can live." What those warps are doing is separate, and you need DRAM / compute / tensor-pipe SOL % + stall distribution to see it.

(b) A ncu stall distribution is a kernel's fingerprint

• long_scoreboard dominant = waiting on DRAM / L2 — memory-bound. Fix: access pattern, tiling.
• math_pipe_throttle dominant = tensor / FP pipe saturated — compute-bound (healthy). Fix: "hard to go faster" — change the algorithm or the hardware.
• wait dominant = MMA output dependency — pipelining deficit. Fix: raise num_stages, reshape accumulator usage.
• lg_throttle dominant = LSU atomic / misaligned — an algorithm design issue. Fix: redesign (Phase 2's reduction v1 → v4).

Only once this distribution is visible does "what to fix" become clear. Without ncu, you can't make that judgement.

(c) "Bigger tile is faster" is an unreliable intuition

In Phase 3, autotune picked BLOCK_M=128, but that wasn't optimal for this shape on L4. Register pressure drained the warp pool. Small tiles (fewer registers → more resident warps, shorter K/V reuse cycle makes software pipelining easier) vs big tiles (each block reads once and computes more → higher arithmetic intensity) — you don't know without measuring. And autotune picking best-by-wall-time doesn't guarantee that best is actually using the HW fully. Confirm with ncu.

The practical tool chain I took home

1. Run the kernel, measure wall time.
2. nsys timeline → check kernel vs transfer critical path.
3. ncu --set detailed → check SOL % for DRAM / Compute / Tensor pipe.
4. Read stall reason distribution. What's dominant?
5. Prescribe based on dominant stall (see table above).

"Before bragging about the speedup," log at least the DRAM % and the dominant stall. Conversely, no more publishing numbers from a "I don't know why it's fast/slow" state.

Closing — what this session is saying

• Lesson 04's pinning effect is asymmetric — D2H 10×, H2D 2.6×. The nsys timeline's 2-hop memcpy path explains it.
• Lesson 02's reduction v1→v4 218× gap is not occupancy — it's atomic serialization (lg_throttle 31 %).
• Lesson 09's FA leaving 22 % to SDPA comes from register pressure (255 regs → half-occupancy) + MMA dependency (wait 39 %). SDPA is already in the healthy tensor-pipe-throttle regime.

These three interpretations are impossible without profiling tools. And with them, the next iteration's "what to change" actually makes sense. A session with zero new kernels — but one that moved forward the starting point of every kernel-tuning session after this.

# Phase 1 — nsys timeline diff (pinned vs pageable)
./scripts/gcp_run_lesson10_phase1.sh <PROJECT_ID> us-west1-b cuda-l4-dev-lesson10

# Phase 2 — ncu reduction v1 vs v4
./scripts/gcp_run_lesson10_phase2.sh <PROJECT_ID> us-west1-b cuda-l4-dev-lesson10

# Phase 3 — ncu ours vs SDPA
./scripts/gcp_run_lesson10_phase3.sh <PROJECT_ID> us-west1-b cuda-l4-dev-lesson10