AI Engineer28 min
60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood
Conference Presentation, Keynote, Talk
- Core Problem: Data platform teams at Robinhood face a mismatch where Pull Requests (PRs) to production data pipelines arrive at "machine speed" due to AI coding assistants, while safety verification remains constrained to "human speed" and limited contextual knowledge.
- Scope of PRs: The volume includes schema changes, pipeline code modifications, configuration updates, and access grants, which previously piled up on a small group of engineers responsible for production safety.
- Risk Factors: Even PRs that pass CI checks (linter, unit tests) can cause downstream breakage, breach freshness SLAs, or expose sensitive data via unregistered consumers.
- Proposed Solution: Implementation of a hybrid "PR Reviewer" system designed to process PRs at machine speed while maintaining human oversight for high-risk decisions.
- Architectural Strategy:
- Layer 1 (Scripts): Deterministic checks against authoritative sources of truth (e.g., Schema Registry, Catalog) to prove facts; fails with CI red status to block PRs.
- Layer 2 (Agent): An AI agent that reviews code to identify risks scripts cannot see, scoring risk levels to either approve or escalate to a human.
- Human Oversight: Reserved for "protected calls" where the agent escalates, ensuring accountability for sensitive decisions.
- Layer 1 Check Details:
- Data Contracts: Verifies if new schema columns exist upstream as registered in the Schema Registry.
- Tags Compliance: Ensures tables have designated owners and PII is correctly labeled in the catalog.
- Lineage Blast Radius: Walks downstream jobs recorded in the lineage tool to detect breakage risks from schema changes; limited strictly to recorded lineage.
- Performance Tests: Simulates changes in staging with production-scale mock data to validate cluster sizing and SLA adherence before merge.
- Layer 2 Agent Capabilities:
- Code Walkthrough: Analyzes the repository to find unregistered downstream consumers (e.g., jobs reading from pipeline definitions or access grants) that lineage tools missed.
- Reasoning Scope: Can only reason over code within the repo; cannot query live production state or catalog directly; must explicitly state "none found" rather than "none exist" for unrecorded consumers like ad-hoc queries.
- Output Format: Generates specific review comments with reasoning on affected lines, assigns a risk score, and executes a decision to approve or escalate.
- Escalation Criteria: The agent is prohibited from auto-approving:
- Grants involving sensitive data.
- Migrations that alter data intent rather than just mechanics.
- Scenarios where the agent lacks confidence in its verdict.
- Case Study (Schema Widening):
- A PR widening a column from
integertobig_integerpassed all Layer 1 checks because the unregistered downstream job was not in the lineage tool. - The Layer 2 Agent identified the unregistered job via repository search, detected the downstream type mismatch risk, and escalated the PR.
- Human intervention confirmed the job was live, the downstream table was fixed first, and the PR merged without incident.
- A PR widening a column from
- Feedback Loop & Learning:
- Human reviewers provide explicit feedback on agent verdicts (agree/disagree with reasoning) to refine rules.
- Manual weekly rule updates currently translate corrections into new logic (e.g., learning that encrypted columns can be labeled non-sensitive).
- Incident Prevention: A "miss" involving re-tagged sensitive columns (where downstream readers were not updated) led to a new rule requiring codebase searches for readers of columns moving to restricted copies.
- Performance Metrics (as of August 31, 2026):
- Automation Rate: ~60% of data pipeline PRs merged without human review.
- Efficiency: ~75% reduction in manual review effort, saving approximately 132 engineering hours per month.
- Safety: Incidents tied to these PRs decreased by ~30%.
- Error Rates: Miss rate (bad PRs approved) is ~2% (mostly due to rule gaps, not judgment failures); false escalation rates are high due to missing context (e.g., encryption logic).
- Identified Limitations & Future Architecture:
- Current State: Layer 1 and Layer 2 run in parallel without shared state; the agent cannot trigger live Layer 1 checks or see their real-time results.
- Proposed Direction: A unified system with a classifier routing the whole PR, granting the agent access to live Layer 1 checks as callable tools to fetch runtime facts.
- Key Takeaways:
- Prove Facts: Use scripts for deterministic facts to reduce AI costs and increase reliability.
- Flag, Don't Clear: Never allow the AI to clear high-risk categories (sensitive data, intent changes); humans must own these decisions.
- Measure False Escalations: Monitor and minimize false escalations to ensure the system actually reduces review load rather than shifting it.