Why Inference Matters 12x More Than Training | YC Paper Club

Core Inference Economics and Importance

• Inference represents the primary cost bottleneck (COGS) for AI companies, often outweighing training costs due to the long useful life of model weights.
• DeepSeek data indicates a 6:1 ratio of inference FLOPs to training FLOPs annually, potentially rising to 12:1 based on a two-year model lifespan.
• Open models now account for over 50% of tokens on OpenRouter and approximately 1/8th of the traffic of Google's Gemini, yet can be 10x cheaper per token with comparable quality.
• DeepSeek reports an 80% gross margin, attributed to aggressive innovation in inference optimization rather than just model efficiency.

Technical Bottlenecks and Metrics

• Pre-fill vs. Decode: Pre-fill is compute-bound (high MFU, ~50% utilization), whereas decode is memory-bound (low MFU, often <5%), making decode the primary throughput bottleneck.
• Throughput Disparity: Including cache hits, throughput per GPU can reach 1.5 million tokens/sec for pre-fill versus 14,000 tokens/sec for decode.
• KV Cache Retention: Together AI retains KV caches for 72 hours to achieve high cache hit rates, which is critical for agentic workflows involving long context.
• Arithmetic Intensity: LLMs suffer from poor arithmetic intensity compared to RNNs or diffusion models because they frequently reload weights from DRAM for every token generated.
• Key KPIs: Systems optimize for Time to First Token (TTFT), Time per Output Token (TPOT), Throughput (tokens/sec), and Goodput (tokens meeting SLA).

Optimization Strategies by Category

• Model Innovations:
  • Mixture of Experts (MoE): Amortizes inference costs by activating only a subset of parameters per token.
  • SSMs & Linear Attention: Reduces attention complexity by processing sequences without full $O(N^2)$ attention or by compressing the sequence domain.
  • Quantization: Models like Minimax's M3.1 achieve performance matching FP8 originals using NVFP4 (2 bytes) via training-free calibration.
  • Distillation: Larger models are distilled into smaller, faster models for specific tasks to reduce inference costs.
• Kernel & Runtime Innovations:
  • Attention Mechanisms: Radix Attention (for prefix reuse), Flash Attention (for tiling efficiency), and Page Attention (for virtual memory paging) are standard optimizations.
  • Together Compiler: An internal agent-driven loop that automatically optimizes kernels based on real production traffic to improve Model Flop Utilization (MFU).
  • Speculative Decoding: Uses a smaller "draft" model to predict tokens for a larger model, accepting guesses to reduce decode steps.
• System & Harness Innovations:
  • Parallelism: Shift from 8-GPU racks to 72-GPU racks (Grace Blackwell) to enable larger expert parallelism domains and higher arithmetic intensity.
  • Disaggregation: Separating pre-fill and decode workloads onto different GPUs to optimize for their distinct compute/memory profiles.
  • Routing & Waterfalling: Directing simple queries to smaller models and complex tasks to larger ones; Recursive Language Models (RLMs) reduce KV cache usage by processing only necessary context.

Quality Assurance and Monitoring

• Provider Variance: The performance gap between the best and worst providers for the same open model can be larger than the gap between open and closed models.
• Error Propagation: In agent workloads with long horizons (e.g., 15 steps), a 10% per-step deviation can compound to an 80% total failure rate.
• Monitoring Techniques:
  • Tool Verifiers: Public scorecards (e.g., from Moonshot/Minimax) benchmark provider-specific tool-calling accuracy.
  • Log Probes: Cheap, continuous monitoring of next-token probability distributions to detect "silent drifts" in serving stacks.
  • Structural Checks: Monitoring output shape (e.g., infinite loops in DeepSeek serving) without inspecting content.
  • Hardware Attestation: Using Trusted Execution Environments (TEEs) to verify that correct weights and software stacks are running on the hardware.

Specific Company Highlights & Technologies

• Together AI:
  • Operates an "AI-native cloud" focusing on end-to-end inference optimization for open models.
  • Utilizes a "Flappy Bird" loop for internal quality red-teaming to catch quantization failures.
  • Achieved 80% cache hit rates by retaining KV caches for 72 hours.
• Ollama (Jeff Morgan):
  • Shifted from local-only to a cloud aggregation service to handle massive 500B-1T parameter models.
  • Advocates for "routing" across multiple providers to ensure reliability and cost-efficiency.
  • Notes a resurgence of local inference capabilities (e.g., Recursive's Flint) alongside cloud solutions.
• Luminal (Joe Fiotti):
  • Uses a RISC-style compiler with "Equality Saturation" (via Egglog) to search the entire space of equivalent program forms.
  • Decomposes PyTorch operations into ~15 primitive ops to avoid maintaining complex rewrite rules for every model.
  • Employs "no cost model" strategy: profiling candidates on real hardware to select the fastest execution path.
• SGLang (Liang Cheng):
  • Developed Radix Attention and Unified Radix Tree for efficient memory management across shared agent contexts.
  • Implemented Zero Overlap Scheduling to hide CPU-GPU latency, achieving up to 2x throughput on Blackwell GPUs.
  • Introduced HiCache (L1/L2/L3 caching of KV states) to reduce TTFT by 56% for long-context retrieval.
  • Deploys Milis to enable inference-inside-training loops for RL training synchronization.

Future Trends & Architectural Shifts

• Architecture Adoption: Previously "dead end" architectures like SSMs (e.g., in Kimi K3) and Sparse Attention are now standard in top-tier open models.
• Recursive Models: Looping models (running tokens through the same layers multiple times) are gaining traction for efficiency without increasing weight count.
• Hardware Interconnects: The shift from 8-GPU to 72-GPU high-bandwidth racks enables new parallelism strategies that were previously bottlenecked by NVLink bandwidth.
• Compiler as a Service: Compilers are becoming a critical differentiator for monetizing heterogeneous hardware (NVIDIA, AMD, TPUs) and commoditizing compute costs.
• Local-Cloud Hybrid: The future involves dynamic routing between high-performance local models (e.g., on Apple Silicon) and cloud providers based on task complexity and cost.