Conference Presentation, Lecture, Keynote
Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operations
- Major AI companies will continue standardizing massive infrastructure build-outs, including new nuclear power and GPU clusters, while startups face a minimum entry barrier of approximately $1 billion to compete.
- Leading-edge model training is projected to consistently require around $10^{26}$ floating-point operations, reinforcing the persistence of market consolidation where only a few dominant architectures and companies achieve state-of-the-art results after five to ten years of industry shrinkage.
- Compilers will remain essential for managing complex, opaque infrastructure, with operator fusion identified as the primary optimization for reducing memory movement and GPU idle time addressed via CUDA Graphs and code-generation wrappers.
- Significant technical challenges include numerical stability issues with lower-precision operations (FP16, FP8, FP4) leading to FMA-induced NaNs, and pattern-matching compilers struggling to adapt to the weekly emergence of new attention variants.
- Relying solely on compiler automation for optimization is expected to fail due to unpredictability in autovectorization and the inability to match human innovation in parallelism search spaces.
- Distributed training systems will encounter persistent scaling hurdles regarding global synchronizations and parameter sharding, while fault tolerance becomes critical as failure rates escalate from one error every 1.8 hours at 16,000 GPUs to one error every 15 minutes at 131,000 GPUs.
- Future ML systems are predicted to shift toward superior programming models, such as FlexAttention, that allow users to express custom optimizations and fused kernels directly, addressing the insufficiency of monolithic APIs for emerging architectural demands.