AI Engineer14 min
40x More Merged PRs With 2.7x the Engineers — Walden Yan, Cognition
Conference Presentation, Product Demonstration
- Cognition's engineering team achieved a 40x increase in merged Pull Request (PR) throughput over the last 12 months.
- This productivity gain occurred alongside a 2.7x growth in engineering headcount.
- Resulting individual engineer productivity is more than 10x higher compared to one year prior.
- PR quality is reported to be maintained or improved despite the velocity increase.
- The organization restructured its development model to treat agents as primary workers rather than tools.
- Larger teams were dissolved into small pods of 2–3 engineers with vertical responsibility.
- Each pod manages the entire lifecycle of their work, including bug triage, QA, product management, and implementation.
- Engineers now assign tasks to AI agents (Devon) rather than implementing code manually.
- Development workflows were decoupled from traditional IDEs and moved to a Slack-centric, evidence-based model.
- Bug reports received via Slack video are forwarded directly to Devon for fixes.
- Devon returns proof of fixes via video, allowing engineers to verify work without opening an IDE or GitHub.
- The ultimate goal is a development lifecycle where no human interaction with coding interfaces is required.
- Infrastructure was fundamentally redesigned to enable autonomous testing and self-maintenance.
- All services were migrated to run locally via Docker Compose within a single virtual machine (VM).
- Third-party services are fully mocked to ensure agents never touch live production environments.
- The VM environment is rebuilt from scratch every hour to ensure agents always start with a fresh codebase.
- Agents automatically fix broken build snapshots, acting as the first line of response to CI failures.
- Agents autonomously write and update their own testing skills as the codebase evolves.
- Security and testing capabilities were enhanced through cloud-based isolated environments.
- Developers have shifted to using iPads for development to reduce local data leakage risks associated with developer machines.
- The team tested complex multi-agent scenarios, such as agents creating multiple accounts to test Zoom voice memos within a single VM.
- A system of recursive self-improvement was implemented to optimize the software factory itself.
- Agents analyze PR data and process signals daily to propose efficiency improvements.
- Specific optimizations identified include reducing CI node setup times and refining lint scripts to catch 20% more bugs prior to review.
- The operational paradigm is shifting from task-based prompting to goal-based job assignment.
- Engineers are encouraged to provide high-level goals (e.g., "improve P99 latency") rather than step-by-step instructions.
- Agents autonomously generate the necessary telemetry, alerts, and weekly reports to achieve these goals.
- The approach requires making all application metrics measurable, including abstract concepts like onboarding retention.
- Agents were utilized to accelerate hardware optimization and cross-platform application development.
- Agents optimized inference throughput on Nvidia's new Vera Rubin GPUs within one week.
- Throughput performance on these new GPUs was improved to exceed previous GB300 generation capabilities.
- Agents discovered 8–9 distinct strategies to maximize Token Per Minute (TPM), with best results approved by humans.
- An Android application was developed in 44 days by an agent acting as the sole developer.
- The agent maintained parity with the existing iOS app by monitoring codebases, Slack channels, and user feedback.
- On peak days, the agent merged 40–50 PRs daily while conducting live, pixel-perfect QA testing against the iOS version.
- Human intervention was minimized to escalating only the most critical decisions.
- Agents optimized inference throughput on Nvidia's new Vera Rubin GPUs within one week.
- The speaker concludes that team productivity is primarily limited by the ambition of the goals set for AI agents.
- The recommended strategy is to establish closed loops where agents optimize measurable metrics 24/7.