Keynote, Conference Presentation
Atindriyo Sanyal, CEO of Galileo: Making Enterprise Gen AI “Reliable”
- Speaker Context: The presenter, from San Francisco-based AI evaluation and observability platform Galileo, is a former foundational architect for Siri and a founding member of Uber's Michelangelo ML platform.
- Market Position: Galileo serves Fortune 500 clients across financial services, consumer goods, telecommunications, and internet sectors to productionize GenAI agents, RAG, and chatbots.
- Industry Statistic: Citing Salesforce data, approximately 50% of simple RAG-based applications remain in internal warehouses, with only ~20% of multi-step agents reaching production.
- Core Challenge: GenAI systems face a "measurement problem" due to their stochastic, probabilistic nature, making prediction and debugging difficult compared to deterministic software.
- Three Pillars of Reliability: Galileo's framework focuses on identifying nuanced failure patterns, tracing complex dynamic agent paths, and deploying low-latency guardrails.
- Failure Categories:
- RAG Failures: Approximately 90% stem from data quality issues (document quality, retrieval parameters), while 10% result from improper instructions.
- Agentic Failures: Includes tool execution errors, "action advancement" failures (inability to move toward a goal), and tool selection errors due to semantic overlap in tool libraries.
- Safety/Security: Issues include data leakage, prompt injections, and bias, often attributed to poor instruction tuning rather than inherent model flaws.
- Evaluation Strategy: Metrics must be customized to specific architectures, moving beyond point-in-time statistics to adaptive measures that evolve via instruction tuning and fine-tuning.
- Observation Evolution: The speaker introduces "error analysis 2.0," a qualitative-to-quantitative workflow where teams first qualitatively identify failure patterns before applying quantitative metrics for root cause analysis.
- Visualization: Complex multi-agent systems (e.g., healthcare workflows) require graph-based visualizations to understand the "butterfly effect" where minor prompt changes impact downstream outcomes.
- Guardrail Latency: Traditional LLM judges for production guardrails are hindered by latency, cost, rate limits, and context constraints.
- Galileo Luna: The company's proprietary suite of 3B–8B parameter small language models (SLMs) serves as real-time evaluators, solving the accuracy-latency trade-off to achieve sub-200ms inference.
- Technical Advantage: Luna models are fine-tunable, open-weight, and designed for reasoning rather than generation, allowing customers to embed custom PII definitions and logic for low-latency, high-accuracy interception.
- Forward-Looking Statement: The speaker predicts that while LLMs will continue to grow smarter over the next 6–18 months, the pressure to provide the correct information and instructions to these models will intensify.
- Call to Action: Interested parties are invited to contact the speaker directly via the provided email for details on Galileo's integrated observability workflow.