Interview, Podcast
Solving Puzzles in Production with Liora Friedberg
- Jane Street anticipates its trading software will perform as expected in production, with production engineers serving as the primary defense to address issues from alerts or human observations, dedicating approximately 25% to 33% of their time to on-rotation support while the remainder covers longer-term initiatives.
- Production engineers are expected to maintain a broad mental model of interconnected systems, contrasting with software engineers who typically specialize in specific systems, while off-rotation tasks will vary based on individual skills, interests, and team placement.
- High-stakes teams, such as order engines, face risks of significant financial loss, with specific examples citing potential losses of one million dollars within a 10-minute window due to urgent support needs.
- The firm has undergone a two-year shift from SSH-ing into production boxes to centralized log management (CLM) and plans to continue migrating from legacy systems to new standard observability tools, though technical debt cleanup and migration may not always rank at the top of development priorities without deliberate time allocation.
- Training for new employees relies on an apprenticeship model where seniors work alongside juniors for several weeks, alongside "incident simulation" systems like "Training Wheels" and pressure-based exercises using games such as "Overcooked" or "Keep Talking and Nobody Explodes," with stricter training models likely applied to high-stakes support teams.
- Recruitment strategies include proposing production engineering roles to candidates with interdisciplinary backgrounds who applied for software engineering, while identifying opinionated candidates from similar roles at other tech companies as a lateral source, with interviewers focusing on the problem-solving process rather than just final solutions.
- The firm expects to encounter novel trading activities requiring human collaboration for decision-making and will continue developing specialized, high-resolution observability tools, such as detailed packet captures, that differ from common industry standards.
- Automation tools used to filter alerts carry the risk of containing bugs that might silence real issues, while postmortems are expected to include technical takeaways and actionable items to reduce the blast radius of future mistakes.
- The work environment varies in intensity, with some engineers finding high-energy support thrilling while others find it unpleasant or stressful, and the firm acknowledges that cleaning up technical debt often requires explicit time allocation despite not always being the top priority.