Interview, Lecture
How GPT, Claude, and Gemini are actually trained and served – Reiner Pope
- Maddox is a new chip startup founded by Rainer Pope, formerly of Google's TPU architecture team; the speaker (John Mueller Jr.) is an angel investor in the company.
- Fast mode pricing (2.5x speed for 6x price) in models like Claude and Cursor is primarily driven by increasing the inference batch size to amortify fixed memory and compute costs.
- There is a theoretical lower bound on latency determined by the time required to fetch total model weights from memory to the chip, regardless of batch size.
- Latency initially scales weakly with batch size but eventually becomes compute-bound; memory bandwidth bottlenecks (KV cache fetch) dominate at lower batch sizes.
- The optimal batch size for frontier models is approximately 300 multiplied by the sparsity factor (e.g., 3,000 tokens/batch for a 10x sparsity model), balancing weight fetch time against compute time.
- Sparse Mixture of Experts (MoE) allows for larger total parameter counts while maintaining similar active compute, provided the batch size scales to amortize the increased total parameter memory fetch.
- Expert parallelism (distributing experts across GPUs within a single rack) is the preferred parallelism strategy for MoE models due to the "all-to-all" communication pattern matching high-speed NVLink intra-rack bandwidth.
- Inter-rack communication is ~8x slower than intra-rack (NVLink) communication; therefore, scaling an MoE layer across multiple racks creates a significant bottleneck unless using pipeline parallelism.
- Pipeline parallelism (distributing layers across racks) solves memory capacity constraints for model weights but does not reduce the memory required for KV caches, as the number of concurrent sequences in flight increases proportionally with pipeline stages.
- For inference on frontier models, systems typically maximize expert parallelism within a single rack (scale-up domain) and use minimal or no pipeline parallelism, as KV cache memory often exceeds the savings from weight sharding.
- The recent expansion of scale-up domain sizes (e.g., Blackwell's 72-GPU racks) significantly reduces latency by allowing parallel weight fetching across all GPUs, addressing the "memory bandwidth" wall rather than just capacity.
- Current frontier models are overtrained relative to Chinchilla scaling laws by a factor of approximately 100x, with total inference tokens (~200 trillion) roughly equaling pre-training tokens, suggesting an economic equilibrium where training, RL, and inference costs are balanced.
- API pricing for context length (e.g., Gemini 3.1 charging 50% more for >200k tokens) aligns with the inflection point where memory bandwidth cost for KV cache fetches exceeds compute costs.
- Input (pre-fill) tokens are ~5x cheaper than output (decode) tokens because pre-fill is compute-bound, whereas decode is memory-bandwidth limited; longer pre-fills allow memory bandwidth costs to be amortized per token.
- Cache hit pricing (~10x cheaper than cache miss) reflects the economic choice between storing KV caches in memory tiers (HBM vs. Flash vs. Spinning Disk) versus the cost of rematerializing them from scratch.
- The "hold time" of cached data (e.g., 5 minutes vs. 1 hour) likely maps to different memory tiers, where shorter durations utilize faster memory (HBM/DDR) and longer durations utilize slower, cheaper storage (Flash/Spinning Disk).
- Cryptographic protocols and neural networks both utilize "mixing" or scrambling functions, but their optimization goals differ: cryptography seeks to maximize output difference for small input changes (avalanche effect), while neural networks use gradient descent to minimize loss.
- Feistel network constructions from cryptography have been adapted into "Reversible Networks" (RevNets), allowing neural networks to be mathematically inverted without storing intermediate activations.
- Reversible Networks trade compute for memory by rematerializing activations during the backward pass, potentially eliminating the need to store large activation maps in HBM during training.
- The physical constraints of modern GPU racks (power, cooling, cable density) limit the ability to scale up interconnects indefinitely, driving the industry toward larger scale-up domains (72+ GPUs) rather than massive single-switch architectures.
- Sparse attention mechanisms (e.g., DeepSeek) can mitigate the quadratic memory growth of context, but aggressive sparsity risks degrading model quality, creating a "Goldilocks zone" for context length optimization.