Conference Presentation, Fireside Chat, Lecture
DeepSeek, Reasoning Models, and the Future of LLMs
- Future state-of-the-art models are expected to adopt structural techniques derived from DeepSeek, such as Mixture of Experts (MoE) with high expert activation ratios (e.g., 256 experts), Group Relative Policy Optimization (GRPO), Multi-Head Latent Attention (MLA), and "couple rope" positional embeddings, driven by a compounding of previously isolated innovations.
- The industry predicts a shift toward significantly longer inference chains, potentially reaching 10,000 tokens (approx. 20 pages) per answer, which will necessitate roughly 20 times more inference compute than current models and force a workload transition from training to "test time compute."
- Training processes will likely evolve to include a high volume of experimentation and failed runs, where the final successful model represents only a small fraction of the total compute budget; total development costs are expected to be orders of magnitude higher than disclosed training run costs (e.g., $5.5M) due to expenses in data selection and exploration.
- The bottleneck of limited training data quality is expected to be overcome by self-improvement loops using synthetic data, where models generate and verify their own reasoning chains; this approach aims to replace human-generated internet data and reduce reliance on prohibitively expensive human-generated complex traces (e.g., math PhDs).
- The required dataset composition for future models is anticipated to follow a specific split of approximately 600,000 synthetic traces for reasoning and 200,000 human-preference traces for behavior alignment, utilizing a "cold start" strategy of intermediate model traces to align base models.
- Performance gains are predicted to rely primarily on "test time scaling" and reasoning quality rather than model size or pre-training data volume, potentially superseding the traditional "Chinchilla" data-to-model-size ratio, while smaller models (e.g., Llama 7B, 13B, 70B) will achieve performance via distillation from large reasoning models rather than direct reinforcement learning.
- Infrastructure demands will increase significantly to support the 20x rise in inference costs, requiring new hardware architectures or efficiency optimizations like quantization and mixed-precision (fp8) training to make 10,000-token reasoning economically viable for consumer devices and edge hardware.
- The AI landscape is expected to accelerate with open weights serving as a blueprint for competitors, breaking the historical 18-month cycle of single-model dominance and compressing the timeline for innovation cycles as the open-source community iterates on DeepSeek's methodologies.
- A "rule-based verifier" approach will likely become the standard for Reinforcement Learning in verifiable domains like math and code, while human preference data will play a larger role only in later SFT phases for creative tasks, reducing the need for human-in-the-loop correction during initial improvement phases.
- Future model development will be characterized by a standard multi-stage training pipeline (SFT -> RL -> SFT -> RL) and the integration of self-reflection mechanisms as a baseline expectation, allowing models to identify and fix errors mid-generation without immediate human intervention.
- The industry anticipates a bifurcation in applications where top-tier reasoning models handle complex problem solving across industries, while distilled, smaller models deployed on laptops and local hardware will serve most everyday use cases to mitigate high inference costs.