Conference Presentation
Benchmarks vs. Reality: Lessons from 750 Trillion Tokens | Chris Clark, OpenRouter | RAISE 2026
- Inference costs are projected to become the primary or secondary operating expense for most knowledge companies, with tokens shifting from cost of goods sold to operating expense in late 2026 after three years of company operation.
- The market consensus has solidified around a multi-model future following a period of uncertainty regarding Anthropic's dominance, with adoption of Open Weight models varying by region and remaining four to six months behind US frontier labs contrary to early 2026 predictions of immediate takeoff.
- New model adoption cycles are lengthening to require one month or more for user adaptation, such as with the DeepSeek 4 release in April, leading users to retain a specific model once identified rather than switching with every release.
- Frontier-facing agents are expected to subsidize costs until future models enable target margins, as premature cost optimization risks falling behind the moving frontier of intelligence.
- Real-world empirical data and continuous evaluation will serve as the definitive benchmarks for model usage, supplanting static labels, while a predictable timeline exists for frontier capabilities to be down-clutched to open weight models over time.
- Infrastructure strategies involve dynamically routing API requests based on real-time performance data to recover failing tool calls, with the expectation that prices for fixed intelligence points will drop to allow workload migration to cheaper inference options.