AI Engineer13 min
The Agents Were Fine. Everything Around Them Broke. - Joey Wang, Harvey
Conference Presentation
Spectre Platform Scale & Adoption
- Platform intake: >1 agent initiated per minute continuously over the last month.
- User base: ~460 Harvey employees utilized the system last month.
- Output volume: Agents wrote 10,000+ Pull Requests (PRs) in total.
- Usage shift: Expanded beyond code and Slack data queries to unanticipated workflows (e.g., marketing generating storybooks from prototypes).
Strategic Positioning & Architecture
- Competitive stance: Spectre does not compete with local agents (Cursor, Cloud Code, etc.) but acts as the cloud orchestration layer under them.
- Scaling logic: Local agents are capped at 5–10 concurrent runs; cloud infrastructure is required for secrets, real stack testing, alerts, and ticket integration.
- Core philosophy: "Hardness" (the harness surrounding the model) is the primary product, not the model itself.
Workflow Automation & "Follow-up Loops"
- Feature launch: Automated follow-up loops enabled in early October to handle CI failures and auto-merge approved PRs.
- Efficiency metric: 80% (4 out of 5) of agent-generated backend PRs passed through the follow-up loop in October.
- Process outcome: Agents now complete the full lifecycle (write, test, fix, merge) without manual intervention.
- Knowledge management: "Projects" feature allows teams to codify tacit knowledge (flags, testing protocols) for all agents to access.
Interface & Interaction Evolution
- CLI dominance: October saw 6,500 tasks initiated via command line; CLI usage grew faster than the web app.
- User segmentation: Emergence of a "headless" user class initiating hundreds of tasks monthly without accessing the UI.
- Integration examples: Cursor/Cloud Code sessions can delegate coordination to Spectre, splitting tickets into sub-tasks with isolated sandboxes.
- Slack integration: "TragibtDot" tool allows users to draft and send answers to Spectre via DM.
Critical Incident: Resource Contention & Fixes
- Incident 1 (Mid-August): A bug triggered ~900 alerts in 4 hours, launching ~900 agents against a single issue, depleting cluster resources.
- Incident 2 (October): An engineer initiated ~950 tasks via CLI in 3 hours, freezing the platform for all users.
- Root cause identified: Resource contention, not model syncing; Git checkout times averaged >1 minute per repo.
- Infrastructure fixes:
- Implemented deduplication: Repeated alerts now join existing investigations.
- Launched a fair launch queue with priority for active keyboard users.
- Performance result: Task start latency dropped from ~2 seconds to 70ms under load.
- Reliability metrics: Successful completion rates improved from 90% (August) to significantly higher by November 4th despite increased burst loads.
Emerging Bottlenecks & Future Challenges
- Merge Queue Pressure: PR volume surged 60% in a single month (late October).
- CI Latency: Merge times increased from 15–30 minutes to >60 minutes; merge queues frequently disabled due to main branch breakages.
- Automation Growth: 110 users run automated agents; safeguards include auto-shutoff after 5 consecutive failures and budget/expiry caps.
- Current unresolved issue: Managing high-velocity agent PRs without breaking the main branch remains an open challenge.
Forward-Looking Goals
- Per-channel agents: Exploring dedicated agents for every Slack channel to autonomously monitor conversations and initiate work.
- Pre-merge validation: Targeting 20% of PRs to have full automated proof of work before human review (currently 1 in 20).
- Open-ended timeline: Acknowledges potential for new systemic failures; plans to revisit "what breaks" in 6 months.
Key Engineering Takeaways
- Prioritize ownership of post-code workflows (testing, checking, merging) over code generation.
- Ensure every human-actionable task is exposeable via API/CLI for agent invocation.
- Implement strict concurrency caps (queues) for agents, humans, and alerts to prevent resource exhaustion.