Cheap Tokens, Expensive Mistakes: The Real Economics of AI at Scale | RAISE Summit 2026

Current Unit Economics Trend

• Inference gross margins for major AI operators are rising significantly despite expanding compute capacity.
• Anthropic reported its first gross profit in Q2 (June) and expects to exceed $1B in operating profit in Q3.
• OpenAI's total company gross margin has risen to approximately 55%, reaching ~65% when excluding free tier usage.
• Amazon Web Services (AWS) reported a gross margin beat driven by profitability in the Bedrock segment.
• Emerging inference API providers (e.g., Base10, Fireworks) are rapidly increasing gross margins during recent venture funding rounds.
• Pricing power remains strong for token generators due to insatiable demand outstripping current compute supply.

Strategic Shift: From Pricing Power to Cost Efficiency

• Market pressure is shifting toward "token maxing" caps and developer-level cost controls via routing logic.
• Startups are leveraging lower-priced open-weight models to undercut frontier lab pricing while targeting profit via efficiency.
• A specific focus on "inexpensive tokens" is emerging, moving beyond simple price hikes to structural infrastructure optimizations.

Technical Drivers of Cost Reduction

• Context Length Challenge: Agentic workflows now frequently utilize 30k–100k token context windows, often generating only 1k tokens, making pre-fill costs dominant.
• KV Cache Offloading: Storing pre-computed context tokens in system memory or high-capacity storage eliminates the need to recalculate them for subsequent turns.
• Recomputation Elimination: By caching context, providers avoid the quadratic cost increase associated with multi-turn interactions (e.g., 10-to-20 turns requiring 100x recalculations).
• Performance Metrics: Implementing KV cache offload has achieved >95% cache hit rates in agentic workflows, reducing costs significantly compared to vanilla inference.
• Hardware Efficiency: Caching offloads pre-fill tasks from GPUs, drastically increasing GPU utilization and shifting the bottleneck from compute to memory I/O.

Implementation Landscape

• Frontier Labs (e.g., OpenAI, Anthropic): Control both model weights and infrastructure; internal optimizations directly increase their profit margins.
• Inference Providers & Neo-clouds: Small players utilizing open-weight models (often from Europe/China) compete on price by layering storage innovations.
• Infrastructure Comparison:
  • NVIDIA Dynamo (DRAM-based): Effective but limited by DRAM capacity, causing performance degradation under high load.
  • Weka Augmented Memory Grid (NAND-based): Demonstrated ability to run 10x more concurrent sessions on comparable hardware.
  • Throughput Gain: The NAND-based solution generated 7x more tokens across aggregate users.
  • Latency Gain: Achieved a 20x improvement in Time-to-First-Token (TTFT) for multi-round sessions (reducing wait times by 95%).
• Software Layering: Margins are increasingly derived from software innovations (multi-token prediction, async pre-fill caching) rather than just model weights.

Benchmarking and Hardware Insights (Agent X Index)

• Scale: The index runs on $80M worth of diverse compute (NVIDIA, AMD, Google TPUs, Amazon Trainium) using real-world production traffic traces ($5M+ value).
• Key Finding: Hardware selection (GPU vs. GPU) is secondary to storage and memory architecture efficiency.
• AMD vs. NVIDIA: While NVIDIA GPUs are generally faster, AMD GPUs with higher memory capacity sometimes outperform NVIDIA in agentic scenarios.
• Storage Superiority: A GPU paired with an advanced storage solution can outperform a "best-in-class" GPU lacking such storage integration.
• Top Performers: Success is defined by a multivariable optimization of GPUs, storage, memory, and front-end networking rather than a single vendor solution.