Interview, Fireside Chat
Why Testing Is Hard and How to Fix It with Will Wilson
- Will Wilson predicts AI will eventually outpace human mathematical capacity ("run us out of Math"), making the ability to verify software a critical limiting factor in development, particularly as AI code generation accelerates.
- The Antithesis platform is expected to achieve "pretty fast results" for distributed systems within the next year or so, with performance improving further as it scales beyond large distributed architectures.
- Jane Street anticipates pushing the "efficient frontier" between software safety and speed, enabling organizations to either ensure near-zero bugs or significantly accelerate development cycles.
- Investment in Antithesis is viewed as a "superpower" for Jane Street, driven by deep subject matter expertise and a strategy to identify value before it becomes legible to the broader market.
- Will Wilson forecasts the company will grow at a disciplined pace, typically remaining below 30% annual growth, to preserve culture, contrasting this with the "terrifying" nature of doubling every year.
- The speaker expects the industry to evolve such that software engineers no longer need knowledge of deterministic simulation or unit testing, instead relying on an automated "box" for verification.
- Without careful oversight, the use of agentic coding loops poses a risk that non-functional software properties, such as architecture and code cleanliness, will deteriorate.
- Antithesis claims to have identified bugs previously missed by Jane Street, enabling more aggressive, large-scale simulations on already well-tested systems and allowing for "more aggressive larger scale kinds of simulations."
- Recent model advancements, specifically "Opus 4.5," are described as creating favorable conditions where agentic coding becomes "miraculous and useful" for the majority of work.
- The company has executed two major strategic pivots in the last two years: shifting to a "customer service mindset" and reorienting toward AI coding, which was previously believed to be ineffective.
- A "curse" of maintaining a production system with very few bugs is the expectation that users will rely on its stability in the future, raising the stakes and necessitating increased engineering effort to maintain that reliability.
- Will Wilson anticipates a future where current software testing practices are "vanishingly rare," estimating that currently approximately zero percent of real software utilizes them.
- The company designs its interview process to assess "second order knowledge" and a candidate's willingness to admit mistakes, reflecting a culture where "we're all wrong a lot."
- The DC location is expected to maintain high engineer tenure, contrasting with the Bay Area's "meta culture of job hopping every nine months" which hinders long-term investment in personnel.
- Internal transaction costs are projected to decrease due to a "high trust" culture, facilitating execution despite the organization's size.
- The speaker acknowledges that software verification is increasingly constrained by "Amdahl's law," a reality that has become starkly visible in the context of AI development.
- Antithesis is currently "violating the turn halting theorem," signaling ongoing efforts to challenge theoretical boundaries as the company moves "up the hierarchy of theorems."
- Lyapunov exponents for systems like Linux are described as "insanely fast," where changing a single bit of memory can alter the entire system state within tens of microseconds.
- Will Wilson admits he is not a "world-class mathematician," drawing a direct comparison to being unable to become a "world-class basketball player," and suggests he might choose a different path today if given the chance.
- High-performing systems face a future challenge where reliability becomes an asset that requires increased engineering resources to maintain as reliance grows.