LESSON 02 · 2026.04.18 · T4

The true cost of copies — pageable vs pinned

Same kernel, same GPU. I only changed the host-side memory choice and end-to-end time jumped 5.6×. D2H alone jumped 12.7×. We break down the source of that gap with numbers and the page table.

Experiment

I added --pinned / --pageable flags to src/vector_add.cu and branched host-buffer allocation between cudaMallocHost and new float[]. Same GPU path — identical kernel, only host memory changes.

Transfer bandwidth plateau

End-to-end total time

Kernel time is identical in both modes. The gap is 100% on the host-side copy.

Why only D2H is 12.7× — page faults

H2D reads pages the user already touched during init. No fault. Hence 2.7×.

D2H is the opposite. new float[n] only allocates virtual addresses without committing physical pages. When the driver CPU-memcpys from staging → pageable, every page incurs a demand-zero fault. A 1 GB buffer means roughly 262k page faults.

Even when pageable is unavoidable, pre-touch your output buffer. A single memset makes D2H several times faster.

T4 with pinned memory hits PCIe Gen3's effective ceiling

H2D 12.3, D2H 13.1 GB/s. Gen3 x16 theoretical 16 GB/s, effective ~13 GB/s. There's nothing more to squeeze. The three remaining levers are:

• Overlap — streams + async copy to overlap kernel and transfer
• Elimination — persistent device buffer, unified memory, zero-copy
• Reduction — kernel fusion shrinks the round trip itself

This is the background for why vLLM pins its KV cache on the GPU and why operator fusion pays off so well.

Small n is a different world

For n ≤ 2¹⁸, the kernel gets pinned against the launch-overhead floor (~4–7 µs). The "bandwidth %" concept loses meaning. At n = 2¹⁸, apparent bandwidth is 140% of theoretical — a spurious number coming from L2 cache hits. The crossover into true HBM-bound territory sits around n ≈ 2²⁰ (4 MB).