Steeve Morin: Why Google Will Win the AI Arms Race & OpenAI Will Not | E1262

Infrastructure Strategy & Hardware Agnosticism

• ZML operates at the infrastructure layer to run any model on any hardware (NVIDIA, AMD, TPU) without compromise or reliance on specific software stacks like CUDA.
• The speaker predicts that within five years, 95% of AI compute demand will be for inference, while only 5% will be for training.
• Running on AMD hardware can yield a 4x efficiency gain compared to NVIDIA for specific large language models (e.g., 70B parameter models) due to better memory utilization per chip.
• The transition away from NVIDIA dominance is currently hindered by high switching costs, specifically the deep integration of PyTorch with CUDA and the massive scale of NVIDIA's installed base.
• The speaker argues that NVIDIA's market dominance in training stems from its acquisition of Mellanox and the superiority of its InfiniBand interconnects, which are critical for multi-node training but less critical for inference.
• A "bubble" in the H100 market is anticipated to burst, as the chip was priced 5x higher than its predecessor (A100) but delivers only 2x the performance for inference.
• Hyperscalers (Google, Amazon, Microsoft) are currently facing a "trilemma" of products, data, and compute, with Google being the only entity possessing all three vertically.

The Shift from Training to Inference

• Training is defined as a "research" phase prioritizing speed of iteration and throughput ("more is better"), whereas inference is "production" prioritizing reliability and latency ("less is better").
• The primary infrastructure bottleneck in production inference is the lack of efficient auto-scaling, leading companies to over-provision expensive hardware (e.g., keeping 100% capacity idle).
• NVIDIA is currently criticized for not being purpose-built for AI; their GPUs are general-purpose graphics chips repurposed via GPGPU, causing performance issues with massive memory transfers in LLMs.
• Specialized chips like Cerebras and Grok achieve high single-stream throughput by utilizing massive on-chip SRAM to avoid slow external memory transfers, though this technology is currently prohibitively expensive.
• The speaker identifies two emerging use cases that will disrupt the current GPU-centric market: Agents and Reasoning, which require low latency from request start to finish rather than high aggregate throughput.
• "Latent space reasoning" is proposed as the next frontier, where models reason internally without converting thoughts to text tokens, a task GPUs cannot currently scale efficiently due to memory bandwidth limitations.
• Future hardware evolution will move toward "compute-in-memory" architectures (e.g., Rain, Fractile), which perform calculations directly where data resides to eliminate transfer latency.

Market Dynamics, Margins, and Supply Chain

• The speaker highlights a "dirty secret" in the compute economy: NVIDIA sells chips at a ~90% margin, while Google's TPU and Amazon's Trainium (Trinium) offer lower margins but are not commercially successful outside their own ecosystems due to high adoption barriers.
• To achieve true hardware agnosticism, the speaker advocates for a "zero buy-in" model where companies can rent compute from any provider and switch instantly, avoiding massive capital expenditures and long amortization cycles.
• DeepSeek's efficiency was achieved not by new hardware but by architectural optimizations that reduced the compute required per token, proving that constraints drive innovation.
• The speaker believes the "winner-take-all" dynamic will not hold for inference; instead, a diversified mix of GPUs, TPUs, and specialized ASICs will coexist based on cost and specific workload needs.
• Export regulations currently hinder Chinese competitors (like DeepSeek) from buying top-tier NVIDIA chips, forcing them to innovate more efficiently, though the speaker predicts they will eventually catch up in software and architecture.

Future Outlook & Risks

• The speaker forecasts a shift in the AI landscape driven by the transition from throughput-bound models to latency-bound reasoning, necessitating new silicon architectures that are not transformers.
• Large monolithic models are becoming inefficient; the trend will shift toward smaller, specialized models or "world models" (energy-based models) that understand physics and causality rather than just text prediction.
• Synthetic data is viewed as a double-edged sword: while effective for coding tasks (where execution validates output), injecting synthetic data for general language models risks model deterioration due to data degradation.
• The "Stargate" $500 billion investment is dismissed as excessive vertical scaling that fails to address fundamental efficiency limits and energy constraints.
• NVIDIA's immediate risk lies in the "downslope" caused by cancelled orders for their Blackwell chips due to yield issues, heat dissipation problems, and physical warping of the chip surface.
• The speaker advises AI startups to avoid reselling compute as a business model, as 98% of the value is captured by hardware margins; instead, startups should verticalize on product and application layers.
• Google is identified as the "sleeping giant" in the AI space, possessing the unique combination of products, data, and compute to dominate the inference market if they execute their internal strategies effectively.