AI Engineer28 min
60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood
Conference Presentation, Keynote, Talk
- The reviewer system aims to handle pull requests at machine speed, utilizing a hybrid architecture where Layer 1 employs deterministic scripts to verify authoritative sources like schema registries, automatically failing checks to block CI for factual errors.
- An AI agent will focus on judgment-based risks that deterministic scripts cannot detect, scoring PRs to either approve changes or escalate them to human reviewers, specifically targeting categories like sensitive data grants or intent-changing migrations which always require human oversight.
- Projected outcomes by August 31st, 2026, include approximately 60% of data pipeline PRs merging without human review, a 75% reduction in manual review effort, roughly 132 saved engineering hours monthly, and a 30% decrease in incidents tied to these PRs.
- Future development envisions a single entry point where a classifier determines whether to review changes individually or collectively, allowing the agent to use repository tools to fetch live facts for judgment while maintaining honesty about search scope limits.
- The team expects to utilize missed corner cases and false escalations to generate new rules for agent guidance, with a proposed but unbuilt plan to let an incident investigation agent assist in weekly rule updates without automatic self-retraining.