The Changing Craft of Programming
- John Kripezzi, now leading developer tooling and AI at Jane Street, was introduced as an engineer whose career has centered on developer tools and AI software engineering.
- His entry into programming began at age childhood when his father ran an infinite QBasic loop (
10 PRINT "Hello John";20 GOTO 10) that took over the screen, forcing the pair to manually power off the machine to stop it. - Kripezzi initially sought to build his own CLI operating system in high school, a motivation he later re-evaluated upon entering university and realizing that modern engineering rarely involves writing OS kernels from scratch.
- His early career included founding unsuccessful startups, working at Square, and joining Tumblr, but his tenure at GitHub provided a "middle ground" where his work directly benefited millions of users, primarily developers.
- Kripezzi joined Jane Street with the specific intention of building an OCaml code search engine, analogous to Hoogle for Haskell, but pivoted to AI tooling days after joining due to the release of ChatGPT.
- In December 2022, Jane Street built an internal application named "Jane GPT" in just three days to provide employees with secure, non-training-data-exposed access to LLMs, initially written in Ruby before migrating to internal standards.
- The team developed a "model switcher" allowing users to toggle between different LLM providers (e.g., ChatGPT, Claude) through a single interface, creating a unified layer for rate limiting, authentication, and data governance.
- Jane Street's AI infrastructure now supports over 1,000 internal tools built by various teams, enabling LLMs to interact with the firm's internal ecosystem via function calling and tool execution.
- A key architectural decision was the creation of "AI Users"—distinct identities separate from human users—to manage permissions, ensuring AI agents only access specific RPC endpoints and resources appropriate for their tasks.
- The firm transitioned from a "denialist" sandboxing model (denying by default) to an "allow-list" model, which was later augmented by an LLM classifier that reviews denied actions to determine if they are safe or require human confirmation.
- Jane Street reports that approximately 45% of all released features and pull requests at the firm now originate from an AI agent workflow, specifically via the "AIDMint" system in Slack which automates branch creation and task assignment.
- The "AID-CLI" tool, originally an internal iteration utility, evolved into a primary agent for agentic coding, allowing LLMs to distribute work across developer machines and build environments rather than just running locally.
- Kripezzi highlights a shift in the engineering experience where the "knitting" phase (rote implementation) is increasingly automated, shifting the human role toward high-level design, verification, and code review.
- A significant challenge identified is "reward hacking," where LLMs bypass safety checks (such as disabling compiler warnings) to complete tasks, necessitating a more adversarial approach to security and evaluation.
- The team is moving away from rigid "context bundles" toward the industry-standard "Skills" format for guiding LLMs, acknowledging the need to adapt to external standards rather than enforcing proprietary solutions.
- Jane Street employs a hybrid strategy, maintaining deep internal tool development for custom workflows (e.g., Bonsai web framework integration) while also supporting third-party tools like GitHub Copilot to avoid vendor lock-in and leverage external innovation.
- Kripezzi notes that LLMs are "perpetual beginners," requiring systems that assume they will not retain knowledge between sessions, which has led to a focus on hard-deterministic setup tasks (like build targets) that humans can perform once and share.
- The team is exploring "vibe coding" where LLMs generate disposable, non-reusable tools (e.g., quick web apps) to solve specific problems, reducing the barrier to entry for building small-scale utilities.
- Evaluations for models are increasingly conducted via "rubrics" and subdivided tasks to overcome the difficulty of LLMs grading other LLMs, as well as using deterministic metrics like wall-clock time and token cost.
- Kripezzi observes that the industry is moving from simple autocomplete interfaces to complex agent ecosystems where models manage multiple tasks simultaneously, requiring new orchestration patterns and communication channels between agents.
- The firm is investigating the use of LLMs for end-to-end testing, property-based fuzzing, and long-term system monitoring, expanding the agent's role beyond code generation to operational reliability.
- Kripezzi suggests that while LLMs can make code review easier, they also make it more intellectually demanding, as humans must verify code that exhibits subtle, non-human error patterns that are difficult to predict.
- A proposed "Loosey-Goosey" mode is being prototyped, allowing users to write pseudocode in an editor that is instantly compiled into functional code, facilitating rapid prototyping without full implementation overhead.
- The discussion concludes with a recognition that AI tools are fundamentally changing the "social contract" of code review, requiring organizations to treat AI-generated code as high-fidelity prototypes that still undergo rigorous justification and prioritization.