newsfilter
← Interviews
AI Engineer13 min

The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey

Joey Wang, Harvey

Conference Presentation
  • Spectre Platform Scale & Adoption

    • Platform intake: >1 agent initiated per minute continuously over the last month.
    • User base: ~460 Harvey employees utilized the system last month.
    • Output volume: Agents wrote 10,000+ Pull Requests (PRs) in total.
    • Usage shift: Expanded beyond code and Slack data queries to unanticipated workflows (e.g., marketing generating storybooks from prototypes).
  • Strategic Positioning & Architecture

    • Competitive stance: Spectre does not compete with local agents (Cursor, Cloud Code, etc.) but acts as the cloud orchestration layer under them.
    • Scaling logic: Local agents are capped at 5–10 concurrent runs; cloud infrastructure is required for secrets, real stack testing, alerts, and ticket integration.
    • Core philosophy: "Hardness" (the harness surrounding the model) is the primary product, not the model itself.
  • Workflow Automation & "Follow-up Loops"

    • Feature launch: Automated follow-up loops enabled in early October to handle CI failures and auto-merge approved PRs.
    • Efficiency metric: 80% (4 out of 5) of agent-generated backend PRs passed through the follow-up loop in October.
    • Process outcome: Agents now complete the full lifecycle (write, test, fix, merge) without manual intervention.
    • Knowledge management: "Projects" feature allows teams to codify tacit knowledge (flags, testing protocols) for all agents to access.
  • Interface & Interaction Evolution

    • CLI dominance: October saw 6,500 tasks initiated via command line; CLI usage grew faster than the web app.
    • User segmentation: Emergence of a "headless" user class initiating hundreds of tasks monthly without accessing the UI.
    • Integration examples: Cursor/Cloud Code sessions can delegate coordination to Spectre, splitting tickets into sub-tasks with isolated sandboxes.
    • Slack integration: "TragibtDot" tool allows users to draft and send answers to Spectre via DM.
  • Critical Incident: Resource Contention & Fixes

    • Incident 1 (Mid-August): A bug triggered ~900 alerts in 4 hours, launching ~900 agents against a single issue, depleting cluster resources.
    • Incident 2 (October): An engineer initiated ~950 tasks via CLI in 3 hours, freezing the platform for all users.
    • Root cause identified: Resource contention, not model syncing; Git checkout times averaged >1 minute per repo.
    • Infrastructure fixes:
      • Implemented deduplication: Repeated alerts now join existing investigations.
      • Launched a fair launch queue with priority for active keyboard users.
      • Performance result: Task start latency dropped from ~2 seconds to 70ms under load.
    • Reliability metrics: Successful completion rates improved from 90% (August) to significantly higher by November 4th despite increased burst loads.
  • Emerging Bottlenecks & Future Challenges

    • Merge Queue Pressure: PR volume surged 60% in a single month (late October).
    • CI Latency: Merge times increased from 15–30 minutes to >60 minutes; merge queues frequently disabled due to main branch breakages.
    • Automation Growth: 110 users run automated agents; safeguards include auto-shutoff after 5 consecutive failures and budget/expiry caps.
    • Current unresolved issue: Managing high-velocity agent PRs without breaking the main branch remains an open challenge.
  • Forward-Looking Goals

    • Per-channel agents: Exploring dedicated agents for every Slack channel to autonomously monitor conversations and initiate work.
    • Pre-merge validation: Targeting 20% of PRs to have full automated proof of work before human review (currently 1 in 20).
    • Open-ended timeline: Acknowledges potential for new systemic failures; plans to revisit "what breaks" in 6 months.
  • Key Engineering Takeaways

    • Prioritize ownership of post-code workflows (testing, checking, merging) over code generation.
    • Ensure every human-actionable task is exposeable via API/CLI for agent invocation.
    • Implement strict concurrency caps (queues) for agents, humans, and alerts to prevent resource exhaustion.