newsfilter
← Interviews
AI Engineer28 min

60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood

Chetas Joshi

Conference Presentation, Keynote, Talk
  • Core Problem: Data platform teams at Robinhood face a mismatch where Pull Requests (PRs) to production data pipelines arrive at "machine speed" due to AI coding assistants, while safety verification remains constrained to "human speed" and limited contextual knowledge.
  • Scope of PRs: The volume includes schema changes, pipeline code modifications, configuration updates, and access grants, which previously piled up on a small group of engineers responsible for production safety.
  • Risk Factors: Even PRs that pass CI checks (linter, unit tests) can cause downstream breakage, breach freshness SLAs, or expose sensitive data via unregistered consumers.
  • Proposed Solution: Implementation of a hybrid "PR Reviewer" system designed to process PRs at machine speed while maintaining human oversight for high-risk decisions.
  • Architectural Strategy:
    • Layer 1 (Scripts): Deterministic checks against authoritative sources of truth (e.g., Schema Registry, Catalog) to prove facts; fails with CI red status to block PRs.
    • Layer 2 (Agent): An AI agent that reviews code to identify risks scripts cannot see, scoring risk levels to either approve or escalate to a human.
    • Human Oversight: Reserved for "protected calls" where the agent escalates, ensuring accountability for sensitive decisions.
  • Layer 1 Check Details:
    • Data Contracts: Verifies if new schema columns exist upstream as registered in the Schema Registry.
    • Tags Compliance: Ensures tables have designated owners and PII is correctly labeled in the catalog.
    • Lineage Blast Radius: Walks downstream jobs recorded in the lineage tool to detect breakage risks from schema changes; limited strictly to recorded lineage.
    • Performance Tests: Simulates changes in staging with production-scale mock data to validate cluster sizing and SLA adherence before merge.
  • Layer 2 Agent Capabilities:
    • Code Walkthrough: Analyzes the repository to find unregistered downstream consumers (e.g., jobs reading from pipeline definitions or access grants) that lineage tools missed.
    • Reasoning Scope: Can only reason over code within the repo; cannot query live production state or catalog directly; must explicitly state "none found" rather than "none exist" for unrecorded consumers like ad-hoc queries.
    • Output Format: Generates specific review comments with reasoning on affected lines, assigns a risk score, and executes a decision to approve or escalate.
  • Escalation Criteria: The agent is prohibited from auto-approving:
    • Grants involving sensitive data.
    • Migrations that alter data intent rather than just mechanics.
    • Scenarios where the agent lacks confidence in its verdict.
  • Case Study (Schema Widening):
    • A PR widening a column from integer to big_integer passed all Layer 1 checks because the unregistered downstream job was not in the lineage tool.
    • The Layer 2 Agent identified the unregistered job via repository search, detected the downstream type mismatch risk, and escalated the PR.
    • Human intervention confirmed the job was live, the downstream table was fixed first, and the PR merged without incident.
  • Feedback Loop & Learning:
    • Human reviewers provide explicit feedback on agent verdicts (agree/disagree with reasoning) to refine rules.
    • Manual weekly rule updates currently translate corrections into new logic (e.g., learning that encrypted columns can be labeled non-sensitive).
    • Incident Prevention: A "miss" involving re-tagged sensitive columns (where downstream readers were not updated) led to a new rule requiring codebase searches for readers of columns moving to restricted copies.
  • Performance Metrics (as of August 31, 2026):
    • Automation Rate: ~60% of data pipeline PRs merged without human review.
    • Efficiency: ~75% reduction in manual review effort, saving approximately 132 engineering hours per month.
    • Safety: Incidents tied to these PRs decreased by ~30%.
    • Error Rates: Miss rate (bad PRs approved) is ~2% (mostly due to rule gaps, not judgment failures); false escalation rates are high due to missing context (e.g., encryption logic).
  • Identified Limitations & Future Architecture:
    • Current State: Layer 1 and Layer 2 run in parallel without shared state; the agent cannot trigger live Layer 1 checks or see their real-time results.
    • Proposed Direction: A unified system with a classifier routing the whole PR, granting the agent access to live Layer 1 checks as callable tools to fetch runtime facts.
  • Key Takeaways:
    • Prove Facts: Use scripts for deterministic facts to reduce AI costs and increase reliability.
    • Flag, Don't Clear: Never allow the AI to clear high-risk categories (sensitive data, intent changes); humans must own these decisions.
    • Measure False Escalations: Monitor and minimize false escalations to ensure the system actually reduces review load rather than shifting it.