AI Engineer16 min
Give Your On-Call Its Own AI Teammate - Swaapnika Motapothula, Chinmay Gandhi & Vlad Tsvang, Meta
Swaapnika Motapothula, Chinmay Gandhi, Vlad Tsvang, Swapnika
Conference Presentation, Keynote
Context & Problem:
- Approximately three-quarters of new code at Meta is now AI-generated, and AI-generated changes frequently cause production failures.
- Operational on-call work has become the primary bottleneck, as the speed of code generation has outpaced the speed of incident response.
- Previous automation attempts failed by creating isolated silos (separate scripts for alerts, triage, deploys, and fixes) that lacked context continuity, forcing human engineers to act as the "glue" between systems.
Solution Architecture:
- Implemented a single "AI teammate" per on-call rotation that operates as one agent across all signals, maintaining a unified team memory and identity.
- The core mechanism is a generic agent loop (model + tools + environment + feedback), protected by a human-in-the-loop governance model.
- The system functions as a "Team Memory" that learns from every shift, allowing knowledge and decisions to transfer seamlessly between on-call engineers.
Autonomy Ladder Framework:
- Level 1 (Shadow): The agent investigates and suggests root causes but takes no action in production.
- Level 2 (Pair): The agent drafts fixes and test plans; execution requires human approval via standard review and CI.
- Level 3 (Autonomous): The agent autonomously handles low-criticality work (e.g., noisy alerts, routine fixes) and only escalates high-criticality issues to humans.
- Earning Mechanism: Agents advance levels only when mistakes are low-cost, easily caught by tests, and the agent has a proven track record on specific actions.
Operational Workflow & User Interface:
- Morning Digests: Agents post summaries at 9:00 AM listing items handled versus those requiring human attention (e.g., "23 items in, 3 need you").
- One-Click Actions: Human intervention is reduced to specific, clear decision points with "run this" buttons for approvals, such as quota bumps or rollbacks.
- Chat Integration: Agents are accessible via chat threads where users can request evidence, ask follow-up questions, or approve fixes without opening dashboards.
- Context Preservation: All investigation history, root cause analysis, and evidence are pre-populated before human review, eliminating the need to re-verify facts.
Shared Agent Identity & Memory:
- The agent belongs to the rotation, not the individual, ensuring that insights (e.g., "roll back config after checkout timeout") persist across shift handoffs.
- Memory is dynamic: semantic retrieval scores accuracy, incorrect items automatically reopen for correction, and valid corrections are promoted or old entries are retired.
- Teams can integrate their own existing agents, runbooks, and skills ("Bring Your Own Agent") without forking the core system.
Results & Observations:
- Workload Reduction: In tested scenarios, the agent handled the "long tail" of routine incidents, reducing human queue items from 23 to 3, though high-criticality decisions remain human-owned.
- Job Shift: On-call engineers transitioned from driving deep investigations to reviewing pre-vetted hypotheses and making final decisions.
- Safety: No production changes are applied automatically without human approval; the agent serves as a review queue, not an autonomous operator for critical incidents.
- Feedback Loop: Every correction or rejection of a draft is ingested as training data to improve future memory retrieval.
Future Roadmap:
- Autonomous Mitigations: Evolving to approve and execute mitigations rather than just drafting them, with a grading system to track agent performance.
- Skill Generation: Enabling teams to convert runbooks into agent skills directly via chat commands without code.
- Rigorous Measurement: Launching formal studies to quantify on-call time saved, moving beyond usage counts to impact analysis.
- Expanded Autonomy: Targeting medium-criticality autonomy in the next phase, contingent on evidence of safety and efficacy.