Interview, Fireside Chat
Sierra co-founder Clay Bavor on Making Customer-Facing AI Agents Delightful
- Core Thesis: The most significant insight from the past 18 months of AI development is that "the solution to many problems with AI is more AI," specifically by using additional models to detect errors in their own output more effectively than they can prevent them initially.
- Founder Background: Clay Bavore (formerly Clay Bevor) spent 18 years at Google, leading Google Labs, AR/VR efforts, and AI applications like Google Lens, before co-founding Sierra in December 2022 with former colleague Brett Taylor.
- Sierra's Mission: The company builds branded AI agents for businesses, enabling them to interact with customers via natural language for tasks ranging from customer service to commerce, effectively replacing hierarchical navigation with conversational interfaces.
- Early Conviction: The founders identified the transformative potential of Large Language Models (LLMs) in mid-2022, citing specific moments such as PaLM-2's ability to explain the 2008 financial crisis using Inception as a metaphor and DALL-E's generation of "avocado chairs" as proof of advanced semantic understanding.
- Platform Capabilities: Sierra's platform currently enables companies like Weight Watchers, Sonos, and Olukai to resolve over 70% of incoming customer inquiries autonomously, achieving customer satisfaction (CSAT) scores of 4.5 out of 5, often surpassing human agents.
- Technological Evolution:
- Past (18 months ago): AI agents were brittle, rule-based systems incapable of handling complex, multi-turn conversations or acting safely.
- Present: Agents can now perform complex tasks like troubleshooting hardware, processing returns, and managing subscriptions with high reliability.
- Future (18 months out): The focus will shift to multimodal capabilities (handling 80% of voice inquiries), deeper reasoning, and solving the remaining "hallucination" and safety challenges.
- AgentOS Architecture: Sierra developed a proprietary "AgentOS" to create industrial-grade agents, addressing three primary LLM limitations:
- Hallucinations: Mitigated through supervision layers where "supervisor agents" review output for factual accuracy and policy compliance.
- Data Silos: Solved by securely integrating agents with real-time systems of record (e.g., order management, CRM) using the Agent SDK, a declarative language for defining goals, guardrails, and workflows.
- Complex Logic: Addressed by scaffolding complex processes (e.g., multi-step returns) outside the LLM context to ensure accurate function calling and parameter usage.
- Tau Bench: Sierra released "Tool Agent User Benchmark" (TAU Bench), a research paper and metric demonstrating that standalone LLMs perform poorly on complex, multi-step tasks (e.g., only 61% success rate on returns), proving the necessity of sophisticated agent architectures over raw model size.
- Pricing Model: Sierra utilizes "resolution-based" pricing, charging clients only when a customer's problem is fully resolved, aligning incentives between Sierra and its clients to maximize both cost savings and customer satisfaction.
- Deployment Process: Sierra treats agent creation as a product design exercise, employing a "deployment team" that works with client CX and marketing teams to embed brand voice, values, and nuanced business policies (including "policies behind policies") into the agent's behavior.
- Experience Manager: A command-center tool for client CX teams to analyze conversation analytics, identify trending issues, and coach agents via feedback loops that automatically generate new tests to prevent future errors.
- 5-Year Outlook: Bavore anticipates AI will act as a "force multiplier" for human creativity and business operations, predicting that feature-length films may soon be entirely AI-generated and that individuals will use AI to rapidly manifest ideas (e.g., coding games, writing songs) in seconds.