Nvidia CTO Michael Kagan: Scaling Beyond Moore's Law to Million-GPU Clusters

Strategic Acquisition & Integration

• NVIDIA acquired Mellanox in March 2019 for $7 billion, a deal that Michael Kagan (former Mellanox CTO) notes the market initially doubted but which ultimately proved "the most successful merger in technology history."
• Following the acquisition, NVIDIA's market capitalization grew approximately 45x, from $100 billion to roughly $4.5 trillion over six years.
• The merger expanded NVIDIA's Israeli operations, with the region's manpower growing more than 2x and a new campus planned for the area.
• Kagan emphasizes that the combined culture focuses on "baking a bigger pie" rather than dividing existing markets, citing a recent partnership with Intel to fuse general-purpose and accelerated computing.

Computing Scale & Architecture Evolution

• The AI workload revolution shifted growth from Moore's Law (doubling transistors every ~2 years) to an exponential demand curve where model size doubles every three months, requiring 10x–16x annual performance growth.
• Traditional "scale-up" (adding transistors to a single chip) is no longer sufficient; the industry has moved to "scale-out" where thousands of GPUs form a single, unified fabric via high-speed interconnects.
• NVIDIA's current GPU systems are rack-sized "machines" (requiring a forklift to move), where up to 72 GPUs are logically presented to software as a single unit via NVLink.
• This architecture allows applications to be split into millions of pieces across thousands of nodes, reducing a one-second task to one millisecond if communication latency is minimized.

Networking & Interconnect Criticality

• Network performance is the primary bottleneck for clusters exceeding single-node boundaries; high bandwidth alone is insufficient without ultra-low latency and narrow latency distribution (low jitter).
• Without Mellanox's technology, scaling beyond a single node would be limited because splitting jobs across machines requires communication to be hidden behind computation.
• Bluefield DPUs (Data Processing Units): These chips run the data center's operating system and infrastructure software, isolating it from customer application workloads to reduce security vulnerabilities (e.g., side-channel attacks like Meltdown).
• Spectrum X: A new Ethernet switch technology designed to manage congestion across distributed data centers by providing telemetry that allows endpoints to adjust communication patterns without relying on large buffers that cause jitter.

Training vs. Inference Workloads

• Generative AI Impact: Inference demand has surged to match or exceed training demand because every generated token in text/image models requires a full forward pass, unlike traditional "one-shot" inference.
• Workload Phases: Inference consists of a compute-intensive "pre-fill" phase (processing the prompt) and a memory-intensive "decode" phase (generating tokens).
• Hardware Strategy: NVIDIA is developing specific GPU SKUs optimized for pre-fill and decode, though they maintain the same CUDA software interface to allow flexible workload placement.
• Deployment: Inference is shifting to edge devices (mobile) while large-scale data centers handle complex reasoning and multi-model tasks; customers may use separate facilities for training and inference depending on optimization needs.

Data Center Limits & Physical Constraints

• The primary constraint on data center size is now energy consumption and heat dissipation rather than the number of chips; gigawatt-scale data centers are the new standard, with some projects targeting 10 gigawatts.
• Air cooling is being replaced by liquid cooling to enable the necessary compute density.
• Scaling across multiple data centers introduces "speed of light" latency variance; managing congestion across these distances requires advanced software to prevent performance degradation from jitter.

Future Outlook & Philosophy

• Moore's vs. "Huang Plus Kagan" Law: Performance growth now follows an exponent of 10x or more per year, driven by software-hardware co-design rather than just transistor density.
• AI in Science: NVIDIA envisions using AI to simulate complex physics and history (e.g., climate change over 50 years), turning social and historical planning into "experimental science."
• AI as Infrastructure: Kagan compares AI to electricity, suggesting it will fundamentally increase human productivity capacity, allowing leaders to undertake 100x more work than is currently possible.
• Future Hardware: Speculation exists for optical interconnects to further reduce latency, though practical limits regarding heat and concrete deployment speed remain.