Interview, Fireside Chat
Building AI Systems You Can Trust
- Core Problem Identification: The speaker realized that after 15 years of optimizing AI models, the primary barrier to enterprise value is not marginal performance gains (e.g., 0.5% on an eval), but the inability to confidently trust system behavior.
- Optimization Risks: Historical efforts to squeeze out performance often introduced "bad behaviors" or robustness issues, a pattern repeating with current LLMs where high-level performance metrics mask latent undesired behaviors.
- Evolution of AI Definition:
- Machine learning is defined as "the stuff that is now easy" (e.g., spell check).
- AI encompasses the "fun new stuff" (e.g., generative capabilities) until it becomes commoditized.
- Generative AI differs from traditional ML by shifting from atomic classification/regression to interactive, expansive, and agentic workflows.
- Company Trajectory:
- SigOpt: Founded based on PhD research to optimize traditional ML parameters (XGBoost, ConvNets, RL); sold to Intel in 2020.
- Intel Tenure: Led AI and HPC division; managed a team scaling from 25 to 200 people; observed enterprise frustration regarding reliability and consistency.
- Distributional: Founded to address the trust gap by focusing on testing behavioral distributions rather than just optimizing performance.
- Challenges in GenAI vs. Traditional ML:
- Non-determinism: Identical inputs can yield different outputs; slight input changes can cause chaotic, large-scale output shifts.
- Non-stationarity: Systems shift constantly due to upstream changes (e.g., provider infrastructure updates, vector database additions, prompt modifications).
- System Scale: Shift from atomic units (single yes/no) to complex, end-to-end agentic pipelines where errors propagate through multiple steps (model calling model, MCP servers).
- Enterprise Trust Requirements:
- Trust requires reliability and consistency aligned with specific business values, not just raw performance or click-through rates.
- Enterprises must verify system behavior continuously, not just at deployment, to adapt to model updates and environmental changes.
- Distributional's Technical Approach:
- Distributional Testing: Instead of strong estimators for "A is better than B," the platform uses a high volume of "weak estimators" to detect subtle distributional shifts in behavior populations.
- Holistic Monitoring: Tracks not just final outputs, but characteristics of the process (toxicity, reading level, tone, retrieval timestamps, reasoning step duration).
- Root Cause Analysis: Enables correlation of performance drops with specific distributional shifts to identify broken components rather than assuming total system failure.
- Platform Centralization Trends:
- Organizations are shifting from scattered "shadow AI" prototypes (using Scikit-Learn on local laptops) to centralized GenAI platforms.
- Benefits of Centralization: Mitigates security risks (unauthorized API usage), enables cost allocation, and provides a unified interface for testing and logging.
- Developer Incentives: Platforms must offer value beyond gatekeeping, such as automated logging, multi-model routers, and built-in testing to encourage adoption over custom stacks.
- Scale of Support: Enterprise routers may support 30+ distinct models with multiple versions to accommodate specific trade-offs (cost, context window, latency).
- Operational Risks & Examples:
- Hallucination: Systems may invent evidence or satisfy implicit user desires rather than providing accurate answers.
- RAG Degradation: Adding excessive historical data to retrieval corpora can cause systems to prioritize outdated information over recent earnings or specific entities.
- Guardrail Failures: Intermediate system transformations can trigger policy violations for seemingly benign user inputs.
- Prompt Evolution: System prompts act as "Conway's Law" for AI, reflecting organizational culture; nested prompts create a combinatorial mess where adding policies creates conflicts.
- Future Industry Outlook:
- AI Ops: Emergence of dedicated teams responsible for maintaining AI reliability (similar to DevOps) to handle system drift and failures.
- Specialization: Co-evolution between labs and enterprises will drive the creation of models specifically optimized for enterprise needs.
- Tech Debt: Enterprises will face trade-offs between model cost, system prompt complexity, and behavioral consistency as applications scale to production.
- Adoption Strategy:
- Companies should transition from "vibe checks" and small datasets to holistic behavioral testing to scale applications to millions of users.
- Testing allows enterprises to safely refactor prompts, switch models, and manage tech debt with the same confidence as traditional software build/test cycles.