Interview, Fireside Chat, Podcast
How Cursor Trained Composer on Fireworks: Distributed Infrastructure for High-Performance RL
- Future Composer versions are expected to be trained from scratch by Cursor rather than initialized from open-source models, while application companies are predicted to shift toward crafting tailored models to optimize the trade-off between quality, speed, and cost.
- Training methodologies for Composer 2 involve continual pre-training to learn code libraries and world knowledge, followed by reinforcement learning (RL) to refine correct code generation, tool usage, and environmental navigation.
- RL training is anticipated to be more complex than pre-training, requiring orchestration of tens of thousands of GPUs, heterogeneous systems, and asynchronous pipelines that accept a few percent algorithmic staleness to maintain high compute efficiency.
- Infrastructure plans include utilizing four globally distributed clusters, leveraging spare production inference capacity, and employing delta compression to manage one-terabyte model weights, potentially reducing weight syncing time to under a minute and swapping weights with only a 30-second pause.
- Specific technical challenges in MoE models, such as numerical mismatches caused by non-deterministic floating-point arithmetic, are expected to be resolved through custom GPU kernels and router replay techniques that align inference and training to near-zero divergence.
- Real-time RL frequency is predicted to increase as model horizons lengthen, with RL also applied to smaller models like Tab to sharpen behavior distributions and utilizing simulated rollouts (e.g., 16 or 128 attempts) to generate precise signals where online RL offers only one.
- Inference during RL is projected to require roughly one-third of GPU allocation to match theoretical forward pass requirements, with models trained to perform "self-summarization" to extend context beyond finite windows.
- The RL environment is expected to consist of a portable harness, a custom virtual machine state, and a reward component, utilizing bursty systems capable of spinning up 100,000 virtual machines instantly to avoid the inefficiencies of standard containers.
- Risks include models learning to distinguish fake environments from real ones and potentially cheating RL systems, as well as the complexity of approximating outcomes in hard-to-verify domains.
- Product-specific companies are expected to specialize by utilizing their own isolated production environments as the most powerful RL setting, whereas generic model builders must cover all tasks within a single model.