Together AI & SemiAnalysis: In Conversation Together AI's Vision For the Future of AI Infrastructure

Market Context and Strategic Origin

• Together AI was founded in late 2022, emerging from research by SemiAnalysis predicting that hyperscalers would lose the future of computing due to architectural path dependence favoring multi-tenant workloads over AI-specific needs.
• The founders argue that the $1.2 trillion cloud market, projected to triple in size over the next decade, requires a fundamental stack redesign from silicon to pricing to address generative AI workloads.
• Current hyperscaler networking architectures remain significantly inferior to "NeoCloud" deployments for AI, a gap persisting even two years after initial critiques.

Performance Benchmarks and Technical Standards

• Together AI achieves 75% Maximum FLOP Utilization (MFU) on H100 GPUs for training workloads, a significant improvement over the previous 50% industry standard.
• The company's chief scientist, Tri Dao, authored "Flash Attention," which has become the de facto standard for accelerating AI workloads; research is ongoing for Flash Attention 3 and next-generation MLP optimizations.
• Performance gains are not automatic across hardware generations; code optimized for Ampere (A100) runs only 30% faster on Hopper (H100) without kernel rewriting, while unoptimized Blackwell runs can be slower than previous generations.
• The ClusterMax project, launched roughly a year ago, evaluates cloud providers across network performance, security, storage, software usability, and critical GPU reliability metrics.

Workload Economics and Infrastructure Challenges

• A shift toward massive inference workloads is occurring, with enterprise models like DeepSeek requiring hundreds of GPUs (e.g., 144 GPUs) to be efficient, despite costs of $350,000–$400,000 per single-socket server.
• Single-GPU inference is deemed ineffective for leading edge models; efficiency requires aggregating workloads across hundreds of GPUs to maximize tokens-per-dollar.
• Optimization efforts have reduced the cost of serving the DeepSeek model from $8 per million tokens down to 30 cents per million tokens, unlocking new use cases like data processing and FAQs.
• GPU reliability is a primary constraint; systems must be designed to handle frequent hardware failures where GPUs fail while CPU and storage remain highly reliable.

Model Evolution and Future Trajectory

• Together AI is pioneering Reinforcement Learning (RL) to train models without reliance on human-created data, exemplified by the "DeepSuite" model which achieved state-of-the-art status in software engineering via pure RL on Code32.
• The strategy involves creating specific sandbox environments where models interact with their own environments to generate training data relevant to specific business workflows.
• The platform supports Mixture of Experts (MoE) architectures, such as DeepSeek's sparse, wide MoE, where experts are split across individual cards to handle large batch sizes and communication loads.
• Vipul Vaid-Prakash and Dylan Patel anticipate entirely new workloads and optimization surfaces to emerge within the next year that have not yet been invented.