Conference Presentation, Keynote
Making GPUs Actually Fast: A Deep Dive into Training Performance
- Naive CPU-GPU pipelines incur massive performance gaps due to sequential execution and frequent synchronization points, such as copying metrics to the CPU or checking for NaN values, whereas asynchronous execution and pinned memory with
non_blocking=Trueallow CPU data preparation and memory transfers to overlap with GPU computation. - Hardware constraints on the H100 GPU include 132 Streaming Multiprocessors (SMs) that cannot communicate efficiently with each other, necessitating a memory hierarchy strategy that prioritizes shared memory (26 TB/s) over L2 cache (7 TB/s) and global memory (3 TB/s) to maximize throughput.
- Kernel efficiency is limited by launch overheads of 5 microseconds on the GPU side, meaning operations requiring less than one microsecond to execute cause the GPU to be starved, while Tensor cores are utilized to optimize small matrix multiplications for higher teraflops.
- The
torch.compiletool automatically traces execution to enable kernel fusion of memory-bound operations like Layer Norm and ReLU, but it introduces "graph breaks" for unsupported operations and may result in slower performance (80-90% of optimized CUDA kernels) if fusion fails due to reliance on Triton templates. - Custom performance solutions follow a tiered strategy starting with pure PyTorch, moving to the Triton Python DSL for high-level GPU programming (yielding 80-90% of CUDA performance), and reserving direct CUDA C++ kernels for critical path optimizations, such as reducing 20-millisecond sequential tensor sums to 30 microseconds.
- Large tensor transfers from CPU to GPU require synchronization to prevent memory deallocation before completion, whereas copying small tensors like single numbers can proceed asynchronously if the data fits in the buffer, and metrics should ideally remain on the GPU to avoid stalling the training loop.