Interview
The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO
- Expectation that the inference market is significantly larger than previously assumed, a view shifted in the last 12 months, with demand for coding agents exceeding two-to-four-year-old forecasts such that any inference provider, including Fireworks, AWS, and Crusoe, will likely succeed.
- Plans to prioritize "inference" as a primary investment thesis and invest in lower-level infrastructure primitives like Modal, Temporal, Supabase, and Convex, while identifying "vertical agents" with domain-specific viewpoints as a key layer, particularly where AI penetration is low but domain expertise is high.
- Anticipation that "meta-learning" and "memory" capabilities will see accelerated progress, with memory layers becoming common and vertically specific within the next few years, whereas general-purpose chat may remain limited by flatline model improvements for specific tasks.
- Projection that costs will rise due to increased token consumption, driving engineering salary spend on coding tools to potentially match or exceed human salaries, while enterprise priorities for agents rank accuracy first, followed by cost (which has risen above latency concerns) and security.
- Expectation that "harness engineering" will focus on efficiency via prompt caching and routing, with routing logic placed in the harness for vertical agents and in gateways for horizontal tasks, acknowledging that native model features like reflection are still developing but expected to improve within 12 months.
- Assessment that being "first" creates a durable brand monopoly, as seen with Langchain and Cursor, contrasting with the AutoGPT example where speed of execution was insufficient to sustain early entry.
- Outlook that open source models will exist in the next 12 months for narrow tasks like routing or guardrails rather than core agentic loops, while fears regarding data leakage to Chinese models are dismissed if models are hosted independently.
- Identification of risks in the robotics sector where excitement and funding exceed current capabilities, and in the customer support space where investment is deemed uninteresting due to market dynamics.
- Expectation that top-tier VCs like Benchmark and Sequoia act as a "third co-founder" and an operational "machine" respectively, with founders advised to accept valuation discounts for this strategic value despite the fundraising process involving perceived "posturing."
- Recognition of past technical missteps, such as the "100% wrong decision" to skip building a gateway layer a year and a half ago and the difficulty of building "SmithDB," alongside a wish that the speaker knew what harnesses looked like in 2022 to build them earlier.