Oct 17, 2026, 9:59 AM ETInterview

The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO

Langchain engineers currently allocate approximately 30% of their engineering salary to coding tools, a metric that has recently declined despite a concurrent increase in token usage.
The primary driver for controlling this spend involves a three-step process:
- Visibility: Tracking granular usage of coding agents to identify spending patterns.
- Cost Controls: Implementing budget caps with VP-level oversight for overrides.
- Harness Engineering: Optimizing the agent code itself (e.g., prompt caching, dynamic tool routing) to reduce token consumption.
An anecdotal case involved a single misuse scenario (a loop capturing webpage DOMs) that incurred $40,000 in costs within one week, highlighting the need for safeguards against improper usage.
Harrison Chase underestimated the demand for inference, stating that the market is "so big" that "everything just works," suggesting a massive and sustained need for inference infrastructure.

Building reliable agents remains difficult because LLMs are non-deterministic black boxes with infinite input dimensions, leading to cascading errors when multiple model calls interact with tools.
A core principle for Langchain is to "erase the product" regularly, recognizing that as models improve, specific engineering workarounds (like generating JSON for tool calling) become obsolete.
Neutral Harnesses are critical for portability, preventing vendor lock-in where memory and context become tied to a single model provider's API.
Model Routing strategy:
- Vertical/Domain-specific agents: Routing must occur at the harness layer to leverage deep context, memory state, and domain-specific logic.
- Horizontal/General tasks: Routing can occur at the gateway layer (e.g., OpenRouter) for simpler use cases like general coding or chat where deep context is unnecessary.
Post-training open-weight models (e.g., Chinese models) are currently best suited for simpler sub-tasks (classification, extraction, guardrails) rather than driving the core agentic loop, though this may evolve.
Continual learning introduces an "outer loop" that processes agent traces to update context, harness code, or model weights, though current LLMs struggle with the meta-reflection required to distill these learnings.

Vertical Agents are identified as the primary value driver; domain experts who can quantify "good" via custom verifiers and evals will win, rather than generic model providers.
Memory layers are considered a massively unsolved problem that will become ubiquitous in 2–3 years, with solutions being highly domain-specific rather than generic.
Open Source Models will likely achieve parity with US frontier models within 12 months, driven by NVIDIA and ecosystem attention, though they remain primarily for narrower tasks.
Infrastructure Investment Opportunities:
- Inference: A massive, underserved market with high demand regardless of the provider.
- Lower-level Infrastructure: Sandboxes, workflow orchestration (e.g., Temporal, Modal), and databases.
- Vertical Applications: Startups with deep domain expertise where AI penetration is currently low.
Investment Rejections: The customer support space is deemed "wildly uninteresting" due to saturation with well-funded incumbents; robotics is considered too far from a "ChatGPT moment" given form factor struggles.

Being first provides a "durable brand monopoly," but success requires exceptional speed and execution; being first alone is insufficient (citing AutoGPT vs. Langchain).
Top-tier VCs (Benchmark, Sequoia) act as operational partners:
- Benchmark: Focuses on product strategy and company positioning.
- Sequoia: Operates as a machine for operational support, including recruiting and marketing.
Founders are advised to avoid fundraising "games" (posturing for leverage) and prioritize relationships built over time, such as the two-and-a-half-year relationship preceding Langchain's Series B.
Harrison Chase cites SmithDB (Langchain's own database for traces) as the hardest technical build, necessitating the creation of a new database to manage agent state effectively.

Agent-to-Agent Interaction: In 2–3 years, infrastructure will shift to vertical agents interacting with horizontal agents; providers will expose their own vertical agents to capture value and manage query volumes.
Inference Costs: While frontier models get cheaper per unit, total token costs will rise due to the exponential increase in requests from multi-agent systems and agent-to-agent communication.
Enterprise Priorities: Accuracy remains the top concern for developers and enterprises, followed by latency and cost; security is a close fourth.
Organizational Design: A shift toward smaller teams (one to two people) driving large numbers of agents is possible but currently limited by the need for human oversight and communication overhead.
Creative AI: Long-term excitement lies beyond enterprise efficiency, specifically in creative applications (e.g., Suno) that generate novel outputs.