Interview, Fireside Chat
The Quest for Community-Trained Open Source AI Models
- Core Objective: News Research aims to accelerate AI by democratizing access to both the end-user product and the underlying open-source code, ensuring the technology remains a "wide green field" for global innovation.
- Philosophical Stance: The organization rejects the traditional "helpless, harmless assistant" persona, instead training "Hermes" models to be neutrally aligned, allowing users to instruct the AI to adopt any specific worldview or persona without moralizing constraints.
- Key Breakthrough (Distro): The group demonstrated that training highly capable, state-of-the-art AI models is possible using only standard internet connections, removing the need for high-speed co-located GPU interconnects (InfiniBand).
- Bandwidth Reduction: Conservative estimates indicate an 857x reduction in bandwidth requirements compared to traditional training methods, with optimistic potential for 2,000x to 3,000x reductions.
- Methodology Insight: Contrary to the assumption that distributed training requires strict synchronization of all model weights, the Distro method allows individual nodes to train independently and transmit only "key insights" (high-value signals) rather than full model states, creating a "bounded" search space.
- Replicability: The team validated their results by discarding prior implementations and re-running the full pre-training process using the AllenAI "Olmo" framework, successfully reproducing the exact loss curves and performance metrics.
- Historical Context of Research: The team previously pioneered the use of synthetic data for training (Hermes) and the "Yarn" context window extension method, which is now utilized by major models from OpenAI, Meta, and DeepSeek.
- Existential Threat Mitigation: Distro was developed to counter the risk of open-source foundational models becoming unavailable (e.g., if Meta ceases releasing Llama series) by providing a path to replicate state-of-the-art models without relying on a single entity's resources.
- Scalability & Architecture: While current tests utilized H100s, the system is designed to be hardware-agnostic, potentially enabling the training of large models using consumer-grade GPUs (e.g., RTX 4090) and Apple Silicon via a decentralized, fault-tolerant network.
- Future Roadmap:
- Immediate: Release of the Distro paper, source code, and ablation data in October.
- Short-term: Development of a full-stack toolchain to enable community members to coordinate distributed training runs practically.
- Long-term: Targeting the ability for the community to independently train 7B models (feasible now) and 405B+ models (projected for late next year) as engineering hurdles are solved.
- Zeroth-Order Optimization Discovery: Initial experiments with zeroth-order (gradient-free) optimization proved ineffective on current hardware, confirming that backpropagation remains necessary for efficient training, though specialized ASICs for inference-only training could eventually enable gradient-free methods.
- Economic Implications: The shift could disrupt the high-cost data center model, potentially driving demand for high-VRAM consumer GPUs and altering NVIDIA's product roadmap toward chips optimized for lower interconnect reliance rather than maximum inter-GPU bandwidth.
- Community Activation: The project leverages a decentralized network of participants, drawing parallels to "SETI at Home," aiming to mobilize latent compute power from individuals and smaller organizations to collectively train global AI models.