Alex Ratner, CEO & Co-Founder, Snorkel AI: Data Centric AI in the Agentic Era

• Alex, co-founder and CEO of Snorkel.ai and faculty at the University of Washington, frames AI development for agentic systems around the critical necessity of high-quality data operations, specifically data generation, curation, and labeling.
• Snorkel's platform supports data operations for automated and programmatic labeling while integrating human-in-the-loop verification by subject matter experts (SMEs).
• The company serves over seven of the top ten US banks and federal/state agencies on the platform side, and provides expert "data as a service" to leading foundation model developers for model tuning and evaluation.
• The core thesis posits that effective evaluation benchmarks—comprising prompts, rubrics, and verifiers—are the primary toolkit for building specialized agentic systems, with these benchmarks effectively serving as the new product requirements documents.

The Three Ingredients of an Evaluation Benchmark

• Prompt/Data Set:
  • Functions as the "exam questions" defining the specific knowledge or capabilities required of the agent or model.
  • Must reflect the complexity of agentic workflows, including multi-step reasoning, tool selection, planning, and multi-turn human-agent interactions.
  • Snorkel released a realistic, challenging insurance underwriting data set on Hugging Face where leading LLMs currently achieve only 60–80% accuracy with significant inefficiencies; the maximum score for optimal tool use is approximately 35%.
  • Construction of these sets requires simulated enterprise environments (e.g., mock PDFs, SQL databases, risk matrices) authored and reviewed by domain experts like underwriters to ensure realism beyond synthetic generation.
  • Data quality is enforced through a hybrid process involving human SMEs, AI-driven quality control models, and adjudication protocols to prevent "regularization effects" inherent in purely synthetic data generation.
  • Success depends not just on volume but on optimizing the mixture and distribution of topics to match specific business contexts.
• Rubrics:
  • Act as the "grading guidelines" that define the distinction between good and bad responses.
  • Requires close coordination between product managers, business stakeholders, and data science teams to codify product goals into measurable criteria.
  • Rubrics are the operational instantiation of product specifications, translating abstract business needs into concrete evaluation metrics.
• Verifiers/Evaluators:
  • Serve as automated or semi-automated mechanisms to assess model performance against the rubric.
  • Snorkel argues against generic "LLM-as-a-judge" approaches, advocating instead for specialized, programmatic evaluators built through data labeling to ensure accuracy.
  • These evaluators function as the "reward function" required for Reinforcement Learning (RL), where accurate verification signals enable model optimization.

Forward-Looking Implications and Technical Trends

• Reinforcement Learning (RL) Operationalization:
  • The combination of a robust benchmark data set and specialized evaluators provides the complete prerequisite stack for RL, allowing models to be optimized to perform well on specific benchmarks.
  • Snorkel researchers, in collaboration with Stanford, DeepMind, and the University of Wisconsin, demonstrated that programmatic weak supervision to build evaluators yields superior RL results.
  • Specific Result: A 7-billion parameter Qwen model, trained via RL using Snorkel's custom evaluators and insurance benchmark, reportedly outperformed GPT-4.0 and other models on the specific benchmark.
• Strategic Shift:
  • As agentic systems evolve from simple prompting to complex, multi-step automation, the ability to build custom evaluation benchmarks becomes the "only" requirement needed to train and deploy specialized enterprise agents.
  • Future AI development will increasingly rely on "push button" optimization where the quality of the input data and evaluation logic directly dictates system performance.