newsfilter
← Interviews
AI Engineer14 min

40x More Merged PRs With 2.7x the Engineers — Walden Yan, Cognition

Walden Yan

Conference Presentation, Product Demonstration
  • Cognition's engineering team achieved a 40x increase in merged Pull Request (PR) throughput over the last 12 months.
    • This productivity gain occurred alongside a 2.7x growth in engineering headcount.
    • Resulting individual engineer productivity is more than 10x higher compared to one year prior.
    • PR quality is reported to be maintained or improved despite the velocity increase.
  • The organization restructured its development model to treat agents as primary workers rather than tools.
    • Larger teams were dissolved into small pods of 2–3 engineers with vertical responsibility.
    • Each pod manages the entire lifecycle of their work, including bug triage, QA, product management, and implementation.
    • Engineers now assign tasks to AI agents (Devon) rather than implementing code manually.
  • Development workflows were decoupled from traditional IDEs and moved to a Slack-centric, evidence-based model.
    • Bug reports received via Slack video are forwarded directly to Devon for fixes.
    • Devon returns proof of fixes via video, allowing engineers to verify work without opening an IDE or GitHub.
    • The ultimate goal is a development lifecycle where no human interaction with coding interfaces is required.
  • Infrastructure was fundamentally redesigned to enable autonomous testing and self-maintenance.
    • All services were migrated to run locally via Docker Compose within a single virtual machine (VM).
    • Third-party services are fully mocked to ensure agents never touch live production environments.
    • The VM environment is rebuilt from scratch every hour to ensure agents always start with a fresh codebase.
    • Agents automatically fix broken build snapshots, acting as the first line of response to CI failures.
    • Agents autonomously write and update their own testing skills as the codebase evolves.
  • Security and testing capabilities were enhanced through cloud-based isolated environments.
    • Developers have shifted to using iPads for development to reduce local data leakage risks associated with developer machines.
    • The team tested complex multi-agent scenarios, such as agents creating multiple accounts to test Zoom voice memos within a single VM.
  • A system of recursive self-improvement was implemented to optimize the software factory itself.
    • Agents analyze PR data and process signals daily to propose efficiency improvements.
    • Specific optimizations identified include reducing CI node setup times and refining lint scripts to catch 20% more bugs prior to review.
  • The operational paradigm is shifting from task-based prompting to goal-based job assignment.
    • Engineers are encouraged to provide high-level goals (e.g., "improve P99 latency") rather than step-by-step instructions.
    • Agents autonomously generate the necessary telemetry, alerts, and weekly reports to achieve these goals.
    • The approach requires making all application metrics measurable, including abstract concepts like onboarding retention.
  • Agents were utilized to accelerate hardware optimization and cross-platform application development.
    • Agents optimized inference throughput on Nvidia's new Vera Rubin GPUs within one week.
      • Throughput performance on these new GPUs was improved to exceed previous GB300 generation capabilities.
      • Agents discovered 8–9 distinct strategies to maximize Token Per Minute (TPM), with best results approved by humans.
    • An Android application was developed in 44 days by an agent acting as the sole developer.
      • The agent maintained parity with the existing iOS app by monitoring codebases, Slack channels, and user feedback.
      • On peak days, the agent merged 40–50 PRs daily while conducting live, pixel-perfect QA testing against the iOS version.
      • Human intervention was minimized to escalating only the most critical decisions.
  • The speaker concludes that team productivity is primarily limited by the ambition of the goals set for AI agents.
    • The recommended strategy is to establish closed loops where agents optimize measurable metrics 24/7.