Interview, Lecture
How GPT, Claude, and Gemini are actually trained and served – Reiner Pope
- Future discussions will utilize a Blackboard lecture format to explore model architecture, ML topics, and the underlying reasons for AI progress trajectories, specifically focusing on clarifying API prices and cluster-level training details.
- Rainer Pope aims to quantify the relationship between batch size, latency, and cost, predicting that failing to batch users results in economics approximately 1,000 times worse than batching, with optimal batch sizes needing to exceed 300 times the sparsity factor.
- In practice, companies are expected to run batch sizes two to three times larger than the theoretical balance point to mitigate real-world inefficiencies, with typical configurations targeting approximately 2,000 tokens per batch to balance latency and throughput.
- System performance is predicted to transition from compute-limited to memory-limited as context length increases, with an optimal "Goldilocks zone" occurring when memory fetch and compute time slopes match; exceeding this range significantly decreases Model FLOPs Utilization.
- DeepSeek V3 is estimated to have 30 to 37 billion active parameters out of 700 billion total, and while sparse attention architectures scale better than linear context lengths, they require a proportional increase in total parameters (e.g., 64x for 4x active) to maintain quality.
- Hardware economics rely on a stable flops-to-memory bandwidth ratio of approximately 300 across A100, H100, and B100 generations, with HBM read/write ratios estimated at 90% reads due to weight matrix and KV cache access patterns.
- Latency management involves balancing a 20-millisecond batch interval against batch processing time to cap worst-case user request latency at approximately 40 milliseconds, with Hopper generation HBM evacuation taking roughly 20 milliseconds and Rubin generation 15 milliseconds.
- Competitive providers must serve traffic equivalent to at least 1,000 times a single Gemini instance (approx. 128k tokens per second) to remain competitive, while optimal inference strategies for frontier models prioritize maximizing expert parallelism within a single rack over pipeline parallelism.
- Infrastructure constraints such as cable density, power delivery, and cooling limit rack size, with the shift from Hopper to Blackwell enabled by moving from trays to racks to support 72-GPU scale-up domains where all-to-all communication is a bottleneck across racks.
- Pipeline parallelism is expected to be less attractive for modern inference due to sufficient memory capacity within racks like Blackwell, though it reduces per-rack memory requirements for weights and incurs latency costs of a few milliseconds per token across rack-to-rack hops.
- Total cost minimization is predicted to occur when pre-training, RL, and inference costs are roughly equalized at a 33% split each, with RL training inefficiency ranging from a factor of 2 to 6 compared to pre-training due to decode costs and incomplete backward passes.
- Current frontier models are estimated to be overtrained by a factor of approximately 100 relative to Chinchilla-optimal token counts, suggesting that the specific 50% price increase for context lengths over 200k tokens aligns with the crossover point where memory fetch time equals compute time.
- Cost per token calculations for 200k context lengths imply a memory footprint of 1 to 2 kilobytes per token, with pre-fill operations being compute-limited and decode operations being memory-bandwidth limited.
- Economic incentives for caching drive 10x cheaper costs for cache hits versus misses, with 5-minute caches likely stored in HBM and 1-hour caches moving to Flash or spinning disk, given that reading slower memory tiers takes on the order of an hour.
- Speculative decoding or multi-token prediction is identified as a significant effect to be addressed later, alongside a prediction that using neural networks for new cryptographic ciphers is dangerous as 99% of such attempts are broken.
- Reversible networks utilizing Feistel network constructions allow neural networks to be inverted, enabling memory savings by rematerializing activations during the backward pass in exchange for increased compute.