Solving Puzzles in Production with Liora Friedberg

Production Engineering at Jane Street: Role Definition and Scope

• Production engineers focus on the "production layer" of systems trading billions of dollars daily, serving as the first line of defense during the trading day.
• The role explicitly separates live, in-day support from traditional overnight or weekend on-call duties.
• Work is split roughly between 25–33% on rotation (active support) and 67–75% on long-term projects to improve system reliability and reduce future incidents.
• Unlike software engineers who typically specialize deeply in few systems, production engineers maintain a broad mental model of how diverse systems interconnect.

Operational Workflow and Incidents

• Routine Issues: Commonly involve new trade types with unfamiliar field combinations that the system cannot route, requiring collaboration with operations teams to define business logic.
• Incidents: Defined as high-severity events impacting multiple teams, such as a system crash caused by manual trade identifier modification.
• Risk Profiles: Support intensity varies by team; Order Engines teams face urgent, high-energy situations with potential financial loss in minutes, while post-trade support teams have more time for root cause analysis.
• Financial Impact: Most downtime results in opportunity cost (missed trading opportunities) rather than direct monetary loss from system errors.

Tooling, Observability, and Automation

• Alerting System (Oculus): A firm-wide tool for discrete alerts and workflow management; features added include page snoozing and auto-acting on pages.
• Alert Health Analytics: Internal tools analyze alert history to identify "flickering" (transient alerts that close without action) and patterns of excessive snoozing.
• Desensitization Risk: High noise levels lead to "alert fatigue," where engineers ignore critical signals; the goal is maximizing signal-to-noise ratio through rigorous filtering.
• Automation Risks: While automation reduces noise, over-filtering can silence genuine alerts; code reviews and "survival of the fittest" prioritization are used to govern automation quality.
• Observability Stack: Transitioning from ad-hoc SSH log parsing to centralized log management (CLM) and distributed tracing; unique capabilities include nanosecond-accurate packet captures.
• Data Format: The firm utilizes S-expressions (Lisp format) for data storage rather than industry standards like JSON or Protobufs.

Culture, Post-Mortems, and Training

• Blame-Free Culture: Mistakes are expected and encouraged to be reported immediately to mitigate impact; the focus is on system improvement rather than individual punishment.
• Post-Mortem Process: Detailed after-action reviews capture timelines, causes, and resolutions, with a dedicated section for actionable technical and process takeaways.
• Template Tension: Structured templates aid new engineers but risk encouraging "fill-in-the-blank" thinking rather than deep synthesis of lessons learned.
• Training Methods: Due to increased system reliability, teams use incident simulations (choose-your-own-adventure style), video game analogies (e.g., Overcooked), and intentional non-production breakage exercises.
• Onboarding: Includes OCaml bootcamp and specialized production bootcamps, followed by an apprenticeship model where juniors shadow senior engineers during live support.

Recruiting and Candidate Qualities

• Recruiting Pipeline: Candidates often apply for software engineering and are later identified as better fits for production engineering based on interdisciplinary backgrounds.
• Interview Evaluation: Involves realistic debugging scenarios where candidates must investigate synthetic failures using command-line tools and data.
• Key Competencies: Successful candidates demonstrate strong communication skills (bridging technical and business domains), deep debugging instincts, carefulness in production interactions, and the ability to remain calm under stress.
• Role Spectrum: The function ranges from heavy coding (building self-service tools) to process engineering (optimizing workflows and team responses).

Future Outlook and Strategic Goals

• Technical Debt: Active effort is required to migrate legacy systems (SSH log parsing) to modern observability standards to reduce complexity.
• Self-Service Goals: Continued investment in tools that allow non-technical users to resolve common issues, reducing the ticket volume for production engineers.
• Culture of Reflection: Ongoing emphasis on team-level conversations and meetings to synthesize big-picture lessons beyond the written post-mortem document.