Conference Presentation, Fireside Chat, Interview, Keynote
Cheap Tokens, Expensive Mistakes: The Real Economics of AI at Scale | RAISE Summit 2026
- Anthropic projects slightly over $1 billion in operating profit for Q3, with full financial details to be disclosed during its IPO.
- Inference economics are shifting toward prioritizing memory and storage solutions over raw GPU speed, as context offloading becomes critical for handling parameter counts of half a terabyte or more.
- Augmented memory grids are expected to enable a tenfold increase in sessions and a sevenfold increase in tokens while delivering 20 times faster time-to-first-token for multi-round sessions compared to DRAM.
- Systems achieving cache hit rates above 95% in agentic workflows will reduce costs by more than half relative to vanilla open models, with pre-filled token costs dropping from 3x to 4x higher than output tokens to a significantly lower ratio.
- Demand for compute remains insatiable with no current lab possessing sufficient capacity, allowing providers to maintain significant pricing power and higher gross margins despite rising compute costs.
- Future infrastructure must evolve from training-focused designs to handling massive parameter counts and wide EP configurations to accommodate context lengths of 30,000 to 100,000 tokens with minimal generation output.
- Multi-turn workflows exhibit quadratic cost growth, where 10 to 20 turns generate 100 to 400 times more work without token caching, necessitating storage offload to prevent recomputation delays.
- Competitive erosion of AI lab pricing power is anticipated as "token maxing" leads to spending caps and smaller providers utilize low prices on platforms like OpenRouter as a freemium model until margins disappear.
- Smaller open-weight model providers face profitability challenges when charging fractions of the cost, while major inference providers are deploying routers to prioritize cheaper tokens over total volume.
- Optimal performance requires combining top-tier GPUs, such as the GB300, with advanced storage, memory, and front-end networking innovations, as AMD's larger memory capacity may outperform faster NVIDIA GPUs in specific scenarios.
- Innovations like multi-token prediction and context handling will allow inference providers to generate substantial savings to pass to customers or retain as increased margins.
- GPU utilization is currently inefficient due to waiting for pre-calculated tokens; utilizing stored and fetched tokens will significantly raise utilization by eliminating recomputation.
- The "Agent X" benchmark is expected to validate that memory offload handling is more decisive for inference economics than specific chip selection by analyzing real production traffic.