Lianmin Zheng

Why Inference Matters 12x More Than Training | YC Paper Club

Dan Fu, Jeff Morgan, Joe Fioti, Lianmin Zheng, Francois

DeepSeek, Together AI, and other industry leaders are transforming AI economics by shifting focus from training to inference optimization, where compute costs now dominate due to high token volumes and long model lifespans. Through innovations in kernel compilers, disaggregated system architectures, and model-specific techniques like Mixture of Experts and quantization, providers are achieving gross margins near 80% while delivering open models at a fraction of the cost of proprietary alternatives. Strategic advancements in runtime scheduling, KV cache retention, and hardware-at-compiler levels address critical bottlenecks in decode throughput, enabling robust, low-latency inference across hybrid local-cloud environments.