Conference Presentation, Keynote, Panel, Fireside Chat
Why Inference Matters 12x More Than Training | YC Paper Club
- Inference is projected to remain the primary investment focus over the next two years due to a 6x to 12x ratio of inference flops to training flops per weight file life, with market cap growth cited as $220 billion over the last decade.
- Traffic patterns may shift from high-cache scenarios to non-cache environments like new chats, potentially causing significant performance degradation and necessitating repeated pre-fill operations.
- Chip innovations will target the decode loop bottleneck where throughput drops from 1.5 million tokens per second with cache hits to 14,000 tokens per second during decode.
- Model architectures including Mixture of Experts (MoE), State Space Models (SSMs), and multi-headed latent attention are expected to reduce test-time flops through cost amortization or reduced full attention layers.
- Kernel and runtime developments such as Flash, Page, and Radix Attention will apply OS and database theory to GPU tiling and memory paging to improve efficiency.
- Harness innovations involving routing, recursive language models, and compaction aim to reduce KV cache usage and service flops by preventing full context loading.
- Open source AI traffic is expected to grow to 400 trillion tokens monthly, with open models comprising over half of tokens on open routers earlier this year.
- Open source models are anticipated to offer near-identical quality to closed models at up to 10 times lower annual costs.
- Together AI plans to build a cloud infrastructure for pre-training, fine-tuning, and inference to generate more tokens globally.
- Request routing strategies will optimize for KV cache residency to ensure requests are directed to GPUs where the cache is already stored.
- Tokenization for tool calls requires rigorous correctness checks to guarantee accurate internet search and command execution capabilities.
- Scheduling choices for batching and response timing must balance total throughput against user-perceived snappiness, potentially requiring distinct approaches for initial responses versus ongoing chats.
- The pre-fill stage is expected to remain compute-bound with high MFU, while the decode stage remains memory-bound, requiring different handling strategies.
- Together Compiler is slated for near-future release to utilize traffic-driven agents for automatic kernel optimization and a recursive self-improvement loop.
- Scale-up domains connecting 72 GPUs via high bandwidth will support higher volumes by isolating experts on different GPUs, requiring robust node failure remediation.
- Quantization to lower precisions like NVFP4 is expected to increase throughput and reduce memory needs while maintaining quality, contingent on rigorous measurement and red teaming.
- Service quality for open models varies significantly between providers, with disparities among providers for a single open model exceeding the gap between closed and open models.
- Coding agent workloads are highly sensitive to inference quality, where a 10% deviation in a 15-step task can result in an 80% error rate due to compounding errors.
- Monitoring via log probes and distribution tests is projected to be more scalable and cost-effective than full benchmark evaluations for detecting silent serving stack changes.
- Labs are expected to publish their own quantized model versions and consistent serving stacks to ensure API experiences match open weights.
- Future Ollama usage will likely involve local-cloud routing where ultra-capable local models on devices like MacBooks handle tasks previously reserved for cloud inference.
- Luminal's compiler aims to commoditize heterogeneous hardware using equality saturation to search optimal kernels across 80 trillion tokens monthly without large teams.
- Runtime performance gains of 1% to 10% are expected from Luminal's compiler by accepting compile times ranging from one to ten hours in favor of inference speed.
- SG-Lan's unified Radix tree is designed to coordinate memory pools (full token, sliding window, recurrent) to prevent leakage and maximize cache hit rates.
- Overlap scheduling is expected to double throughput on Blackwell GPUs compared to vanilla scheduling by delaying device-to-host synchronization to eliminate GPU bubbles.
- HiCache is projected to reduce Time to First Token (TTFT) by 56% for long-context agent workloads by managing KV cache across L1, L2, L3, and host storage.
- Unified memory pools will replace fixed-ratio pre-allocation to dynamically handle context lengths from 4K to 1M tokens, utilizing compaction to eliminate fragmentation.
- MILEs will enable inference within the training loop through fully synchronized execution for rollouts and training.
- The Astronaut Summit is scheduled to occur next month to discuss these industry advancements.