newsfilter
← Interviews
AI Engineer16 min

Give Your On-Call Its Own AI Teammate - Swaapnika Motapothula, Chinmay Gandhi & Vlad Tsvang, Meta

Swaapnika Motapothula, Chinmay Gandhi, Vlad Tsvang, Swapnika

Conference Presentation, Keynote
  • Context & Problem:

    • Approximately three-quarters of new code at Meta is now AI-generated, and AI-generated changes frequently cause production failures.
    • Operational on-call work has become the primary bottleneck, as the speed of code generation has outpaced the speed of incident response.
    • Previous automation attempts failed by creating isolated silos (separate scripts for alerts, triage, deploys, and fixes) that lacked context continuity, forcing human engineers to act as the "glue" between systems.
  • Solution Architecture:

    • Implemented a single "AI teammate" per on-call rotation that operates as one agent across all signals, maintaining a unified team memory and identity.
    • The core mechanism is a generic agent loop (model + tools + environment + feedback), protected by a human-in-the-loop governance model.
    • The system functions as a "Team Memory" that learns from every shift, allowing knowledge and decisions to transfer seamlessly between on-call engineers.
  • Autonomy Ladder Framework:

    • Level 1 (Shadow): The agent investigates and suggests root causes but takes no action in production.
    • Level 2 (Pair): The agent drafts fixes and test plans; execution requires human approval via standard review and CI.
    • Level 3 (Autonomous): The agent autonomously handles low-criticality work (e.g., noisy alerts, routine fixes) and only escalates high-criticality issues to humans.
    • Earning Mechanism: Agents advance levels only when mistakes are low-cost, easily caught by tests, and the agent has a proven track record on specific actions.
  • Operational Workflow & User Interface:

    • Morning Digests: Agents post summaries at 9:00 AM listing items handled versus those requiring human attention (e.g., "23 items in, 3 need you").
    • One-Click Actions: Human intervention is reduced to specific, clear decision points with "run this" buttons for approvals, such as quota bumps or rollbacks.
    • Chat Integration: Agents are accessible via chat threads where users can request evidence, ask follow-up questions, or approve fixes without opening dashboards.
    • Context Preservation: All investigation history, root cause analysis, and evidence are pre-populated before human review, eliminating the need to re-verify facts.
  • Shared Agent Identity & Memory:

    • The agent belongs to the rotation, not the individual, ensuring that insights (e.g., "roll back config after checkout timeout") persist across shift handoffs.
    • Memory is dynamic: semantic retrieval scores accuracy, incorrect items automatically reopen for correction, and valid corrections are promoted or old entries are retired.
    • Teams can integrate their own existing agents, runbooks, and skills ("Bring Your Own Agent") without forking the core system.
  • Results & Observations:

    • Workload Reduction: In tested scenarios, the agent handled the "long tail" of routine incidents, reducing human queue items from 23 to 3, though high-criticality decisions remain human-owned.
    • Job Shift: On-call engineers transitioned from driving deep investigations to reviewing pre-vetted hypotheses and making final decisions.
    • Safety: No production changes are applied automatically without human approval; the agent serves as a review queue, not an autonomous operator for critical incidents.
    • Feedback Loop: Every correction or rejection of a draft is ingested as training data to improve future memory retrieval.
  • Future Roadmap:

    • Autonomous Mitigations: Evolving to approve and execute mitigations rather than just drafting them, with a grading system to track agent performance.
    • Skill Generation: Enabling teams to convert runbooks into agent skills directly via chat commands without code.
    • Rigorous Measurement: Launching formal studies to quantify on-call time saved, moving beyond usage counts to impact analysis.
    • Expanded Autonomy: Targeting medium-criticality autonomy in the next phase, contingent on evidence of safety and efficacy.