Raph Levien: A Taste of GPU Compute

Hardware Performance Trends

• Single-core CPU performance (Moore's Law/Dennard scaling) has plateaued; high throughput now relies on parallelism, which GPUs provide at roughly 10x the floating-point operations per second of CPUs.
• Throughput per dollar and per watt for GPUs is approximately 10x higher than CPUs across mobile, desktop, and server markets.
• Modern laptop chips (e.g., Ice Lake) dedicate the vast majority of silicon area to GPU execution units, memory caches, and interconnects, treating CPU cores as secondary.
• Execution unit latency is approximately 2 nanoseconds for ALU operations, while global memory access latency ranges from 100 to 300 nanoseconds.
• SIMD lane widths vary by manufacturer: Intel uses 8 or 16, NVIDIA uses 32, and AMD uses 64 lanes per execution thread.

Architectural Hierarchies and Latency

• The hardware hierarchy moves from individual lanes (low latency, ~512 bytes capacity) to SIMD units, execution units, sub-slices, and finally to main memory via slice and memory bus.
• Sub-slices share instruction caches, local thread dispatchers, and texture samplers; memory accesses through these layers incur ~100ns latency.
• Thread group shared memory offers latency and bandwidth superior to global memory but is limited to roughly 64KB per thread group, requiring explicit management.
• Texture samplers provide specialized hardware for linear/bilinear interpolation and L1/L2 caching, returning the industry to a model where memory lookups can be faster than computation.

Programming Models and Functional Approaches

• GPUs prioritize throughput over single-thread latency; performance is achieved by maintaining high occupancy of execution units via thread arbitration.
• Branch divergence occurs when SIMD lanes take different paths; if only a subset of lanes are active, the inactive lanes remain idle, degrading work factor.
• A functional programming model is proposed for GPU compute: decomposing problems into grids of thread groups where outputs are explicit functions of inputs, mapped to execution graphs.
• The proposed 2D renderer maps scene objects to a rectangular grid, using a fold operation (Porter-Duff over operator) to composite pixels in parallel.
• Pure functional parallelism reads all objects for every pixel (high bandwidth, low work factor); sequential iteration using bounding boxes reduces read bandwidth but sacrifices parallelism.
• A hybrid tiling strategy is proposed: a first dispatch filters objects by tile (generating a command list), and a second dispatch renders tiles using the filtered list, achieving a ~256x improvement in read efficiency.

Optimization via Transpose and Bitmasking

• Optimizing bounding box checks involves parallel reads of all bounding boxes per thread group, followed by a matrix transpose of intersection bitmasks.
• This transpose operation allows lanes to determine exactly which objects intersect their assigned tile, enabling the hardware to skip non-intersecting objects entirely.
• This approach minimizes divergence and ensures that memory writes occur only for relevant object-tile pairs, significantly improving work factor.
• The strategy effectively specializes a scene description into bytecode (object lists per tile) before interpretation (rendering), a pattern applicable beyond 2D graphics.

Software Ecosystem and API Evolution

• CUDA remains dominant for machine learning, but industry trend is shifting toward standardizing on Vulkan, DirectX 12, and emerging WebGPU for compute workloads.
• SPIR-V is becoming the standard intermediate language, decoupling source compilation from hardware drivers and enabling cross-vendor toolchains.
• Vulkan 1.1+ and DirectX 12 provide access to modern features like subgroup shuffles/reduces, restricted pointers, and formal memory models (distinguishing visibility and availability).
• WebGPU utilizes WGSL (WebGPU Shading Language), a textual format semantically equivalent to SPIR-V, designed for browser safety and open-source implementation (Mozilla's wgpu, Google's Dawn).
• New language initiatives targeting GPUs include Halide (image processing), Futhark (functional ML), CodeDfns (array language), Julia (interactive ML), and TensorFlow MLR (converting TF graphs to SPIR-V).

Challenges and Future Outlook

• Hardware-specific optimization remains a hurdle; subgroup sizes often differ by GPU generation, requiring manual tuning or complex heuristics rather than purely compiler-driven optimization.
• The industry is moving away from high-level "magic" compilers toward tools that allow developers to explicitly encode hardware mapping strategies with minimal friction.
• Specialized hardware (tensor cores) will increasingly diverge for machine learning, while general-purpose compute APIs (WebGPU/Vulkan) will unify deployment for other domains.
• Standardized test suites and formal memory models (e.g., in Vulkan 1.2 and WebGPU) are critical for ensuring correctness and performance portability across diverse GPU implementations.
• The speaker predicts a migration of workloads from proprietary CUDA ecosystems to open standards like Vulkan and WebGPU as API maturity improves and tooling stabilizes.