Conference Presentation
When to Build Your Own Agent Harness | Harrison Chase, LangChain
- The agent and harness ecosystem is expected to continue significant growth from 2022 levels, with the near-future trajectory suggesting users will increasingly desire ownership over model, context, and harness components to truly own their intelligence.
- While models are projected to reach sufficient capability for basic tasks allowing reliance on off-the-shelf harnesses, users moving further out-of-distribution are predicted to require custom-tuned harnesses or complete custom cognitive architectures to ensure predictability and control, particularly in sectors like financial services.
- LangSmith Engine is planned to operate continuously in the background to curate traces and identify issues, with specific plans to implement a sprint to "Codexify" the engine by incorporating Codex learning into the core harness and eventually running the system on itself with performance reports sent via Slack.
- The company expects to benchmark models and harnesses against the "issue bench" as industry standards evolve, predicting that mission-critical organizations will eventually build their own private benchmarks to define "good" performance within their specific context.
- Strategic plans include leveraging the engine to automatically suggest fixes to prompts, context instructions, or harness code, and utilizing fine-tuned small language models or code to generate synthetic feedback for traces, supporting a system update cycle involving harness engineering, model fine-tuning, or memory upgrades.
- A core expectation is that most agent failures stem from insufficient context rather than model capability alone, driving a strategy where retaining ownership of organization memory, traces, feedback, decisions, and institutional context enables continuous learning loops that compound firm value.
- Market trends anticipate convergence in model coding capabilities, while harnesses may diverge as labs specialize in verticals like bio-agent tasks, with UX designs for agent interfaces noted as currently underestimated but capable of being leveraged to gather user feedback.