Production Engineering When Trading Billions of Dollars a Day

Context & Problem Statement

• Speaker Mark, a production engineer at Jane Street for nearly ten years, defines the core challenge as managing software in an environment prone to hardware failures, network outages, exchange issues, and internal deployment bugs.
• A trading system's primary input is real-time market data, which drives strategy execution via an order engine connected to exchanges, followed by position tracking and regulatory reporting.
• An order consists of five essential components: side (buy/sell), quantity, instrument symbol, price, and currency; errors in any component can be catastrophic.

Key Features of the Trading Production Environment

• Universal Order Importance: Unlike typical web services where 99.99% success rates are acceptable, every single order at Jane Street must succeed; a 0.01% error rate can lead to insolvency.
• Catastrophic Risk of Fat Finger Errors: A December 8, 2005, error by Mizuho (selling 600,000 shares at 1 yen instead of 1 share at 600,000 yen) resulted in a $27 billion loss (27 billion yen) and a 2% drop in Japan's primary stock index.
• Adverse Selection: Trading environments actively punish mistakes; as soon as a bad trade occurs, other market participants instantly execute trades against the firm, accelerating losses.
• Knight Capital Incident (2012): A trading firm lost $450 million in 45 minutes due to a dormant bug triggered by a repurposed configuration flag on a single un-upgraded server instance, leading to the firm's acquisition by a competitor.
• Critical Timing Windows: Trading volume spikes at market open and close, with essential metadata downloads and system startups required within a narrow two-hour window prior to the open to avoid missing the session.
• Internal User Base: Trading firms benefit from high-bandwidth, direct communication with internal users (traders), allowing for precise, non-politicized incident reporting and resolution.

Monitoring & Alerting Strategy

• Rejection of SLO-Based Monitoring: Jane Street avoids traditional Service Level Objectives (e.g., "99.99% success") for live trading systems because they fail to detect the specific, catastrophic 0.01% errors that matter.
• Adoption of Event-Based Alerting: The firm uses code-level checks that force engineers to enumerate and decide on alerts for every specific edge case, ensuring high-fidelity notification of distinct failures.
• Symptom-Based Over Cause-Based: Alerts are designed to trigger on observed symptoms (e.g., 500 errors) rather than specific causes (e.g., database down) to avoid duplicate alerts and alert fatigue during maintenance.
• Orthogonal "Epistemic Health" Alerts: Jane Street utilizes abstract alerts that monitor the correctness of the world model rather than technical health, such as "Fill Too Good" (making too much money) or abnormal market volume spikes.
• Defense in Depth: The firm runs independent, redundant alerting systems written by different teams to ensure that if one detection layer fails due to shared bugs or logic errors, others will catch the issue.
• Monitoring as Critical Infrastructure: Monitoring systems are treated as more robust and reliable than the trading systems they monitor; trading systems are halted if monitoring is unavailable.
• Cultural Obsession with Signal-to-Noise: The firm maintains a culture where traders and engineers collaborate closely to refine alert logic, as noisy alerts are considered worse than useless in a high-stakes environment.

Incident Response & Business Context

• Business Context Necessity: Production engineers require deep domain knowledge (e.g., understanding ticker symbols like "SPU" or specific exchange locations) to resolve incidents rapidly without redundant clarification.
• Sample Incident Analysis: An incident occurred where "Fill Too Good" alerts triggered on multiple symbols, revealing that the order engine was trading on stale market data (previous day's close) due to a missed exchange-driven feed partition change notification received two weeks prior.
• Incident Resolution: The root cause was identified when traders reported stale data, allowing the market data team to confirm a bug in handling exchange notifications; the system was resolved by applying a configuration change to acknowledge the new partition.
• Post-Incident Takeaways: Rapid technical response was insufficient because the damage occurred at market open; the true value lay in the high-fidelity communication between traders and engineers.

Q&A Insights

• Exchange Notifications: Firms must manually parse hundreds of routine exchange notifications to identify the rare few that require action, a process where errors (missed notifications) have historically occurred.
• Alert Implementation Responsibility: Every software engineering team is responsible for designing the event-based alerts within their own codebases, rather than relying on a separate operations team.
• Standardization: While core alerting libraries and logging tools are shared, there is no strict top-down enforcement of terminology conventions across teams, requiring engineers to learn specific team contexts.
• Risk Management Layers: Firm-wide risk is managed through a mix of automated gates and human oversight, where traders monitor aggregate risk and automated systems enforce hard limits on individual desks.
• Cascade Alerting: Event-based alerting mitigates alert storms by focusing on high-level symptoms rather than low-level component failures, preventing thousands of simultaneous alerts when a single issue affects the broader system.