Interview, Fireside Chat
Building AI Systems You Can Trust
- The primary barrier to deriving AI value is expected to remain the inability to trust systems rather than raw performance metrics, a focus on high-level evaluations that currently masks undesired behaviors.
- System confidence and reliability issues are predicted to be pervasive among Fortune 500 and Global 2000 enterprises struggling to scale from prototypes to production due to fears of real-user data exposure.
- AI systems are forecast to evolve from atomic classification tasks to collaborative, agentic architectures where models call other models or external modules, making non-deterministic input changes critical as they propagate into massive behavioral shifts.
- Enterprises are expected to shift from individual science projects to centralized Gen AI platforms to manage scale, costs, and security risks like shadow AI, likely establishing a "gateway or router" to control access to approximately 30 supported models and versions.
- Future infrastructure will involve centralized logging to layer testing and behavioral analysis on top of, enabling developers to characterize output distributions and detect unsupervised changes between "today and yesterday."
- A rise in specialized "AI Ops" teams is predicted to fill the gap where traditional DevOps struggles with unpredictable AI behaviors, alongside a co-evolution where labs adapt foundational models to enterprise needs and industries adjust internal tools.
- Organizations will face specific technical risks including retrieval mechanism failures when historical data is added to vector databases, leading to the retrieval of outdated information.
- Firms will be forced to make trade-offs between cheaper and expensive models or refactor system prompts, which act as a form of "Conway's Law" mirroring internal organizational culture, while building tech debt over six to 12 months of production use.
- Distributional plans to leverage past mistakes from SigOpt and Intel to build a solution 10 times faster that uses a large number of potentially weak estimators to root-cause performance drops by identifying shifts in specific component distributions.
- The company's approach anticipates moving beyond "in distribution" or "out of distribution" concepts to characterize actual distributions of outputs, allowing teams to define "reliable" through specific supervisory preferences after initial detection of changes.
- By providing confidence through testing, enterprises are expected to tackle higher-risk problems involving financial, reputational, or regulatory consequences rather than limiting themselves to low-hanging fruit like internal chatbots.
- The industry will see a rise in specialized models built to solve specific enterprise needs, driven by feedback loops between labs and enterprise adaptation, requiring robust testing to safely refactor system prompts or switch architectures without breaking the system.