Interview, Fireside Chat, Podcast
Factory’s Matan Grinberg and Eno Reyes Unleash the Droids on Software Development | Training Data
- The market is expected to introduce new benchmarks, specifically "SweepBench 2" and "SweepBench 3," within the next two to three years, which will focus on evaluating code for utility and idealness rather than simple correctness, potentially rendering current high benchmarks like 80-90% on SweepBench less relevant before they are reached.
- The specific role of "software engineering" may evolve over five years into titles such as "software curator," "cultivator," or "orchestrator," reflecting a shift where engineers move from low-level implementation to higher-level tasks like architecture and planning, while the total number of people involved in software creation could increase tenfold with individual impact equivalent to 100 to 1,000 people.
- The company's roadmap includes building "physical droids," leveraging the convergence of multimodal function calling LLMs and decreased hardware costs, though foundational model training is viewed as a battle to be outsourced to other labs, with the strategy to first build a user product to earn the right to fine-tune.
- Product value is projected to multiply significantly, with expectations that the product will become "10x better" if "OpenAI releases GPT-6 or 7," as new models from cutting-edge research labs like OpenAI, Anthropic, and Google are anticipated to bring incredible improvements.
- Current and near-future capabilities already allow AI to act as "intern-level engineers" for specific tasks like code review and testing, with some deployments rated as the "best reviewer on their team," and tools like the "test droid" currently saving engineers approximately 40 minutes per day.
- Without adjustments to review processes, the adoption of AI tools may cause code churn to increase by 20 to 40 percent; however, current customer data shows an average cycle time increase of around 22 percent and an average reduction in code churn by 13 percent.
- Success in the market relies on executing faster and maintaining a high level of obsession with the mission, as the ability to build a product that people actually use is deemed a prerequisite for value, with fine-tuning and training unable to compensate for a lack of product-market fit.
- The company defines its main success axes as saving engineering time, increasing speed, and improving code quality, prioritizing "engineering velocity" as the primary metric for leadership rather than metrics like auto-completed acceptance rate which may not align with business objectives.
- The internal "code droid" has demonstrated success in specific integration tasks, such as a GitLab integration, where it fully specified and implemented sub-tickets, saving the team material amounts of time on work previously dreaded for months.
- Innovation at the foundation model layer is predicted to focus on latency, context windows, and performance on subsets, while application layer innovations will drive complex interactions, data flow, and long-term planning capabilities through "droid architecture."
- The speaker warns against viewing the industry as filled with "picks and shovels," noting that few real products currently utilize AI effectively, and suggests that competitive pressure should be viewed as motivating noise if a team is truly mission-obsessed.
- External pressures from competitors are expected to be irrelevant to teams focused on their mission, while the speaker cautions that stress over new product releases in response to model updates may indicate a need to adjust product strategy.
- The threshold for reliable engineer use is already met in production for specific use cases, despite benchmarks having limitations as approximations, and the ability to handle failure trajectories and edit mid-process is considered necessary for immediate productive gains.
- The speaker notes that while documentation covers 95% of use cases, the remaining 5% is critical, and automating enjoyable development parts while leaving engineers with only reviewing, testing, and documenting would be a "depressing hellscape."
- The speaker implies that the industry is in a transition where tools like "test droids" and "review droids" operate at high levels immediately, with complexity slowly climbing as AI capabilities improve and the role of the engineer shifts further toward design and orchestration.
- The company contrasts its strategy of relying on "customers as the best benchmark" against labs that make submissions, aiming to build a product people use rather than focusing solely on benchmark scores which may not test real-world software engineering tasks like feature requests or migrations.