Horace He: Building Machine Learning Systems for a Trillion Trillion Floating Point Operations

Meta compiler engineer Horace He analyzes the dramatic consolidation of AI infrastructure, noting that modern model training now requires massive power resources and billions in capital to achieve state-of-the-art performance. He details how the industry has transitioned from simple imperative execution to complex compiler strategies like FlexAttention and `torch.compile`, which are essential for managing the critical balance between GPU compute and memory movement. Ultimately, He argues that the primary challenge in this field is shifting focus from pure optimization to designing robust programming models that allow developers to reliably express complex performance trade-offs in large-scale distributed systems.