Conference Presentation, Lecture
Production Engineering When Trading Billions of Dollars a Day
- The speaker plans a 15-minute introduction that may be shorter or longer, followed by a sample incident and Q&A session to illustrate technology, processes, and culture built on production environment features.
- The presentation covers high-frequency trading risks, noting that a single logic error (e.g., swapping share price and quantity) in a tight loop could lose approximately 600,000 squared yen, with uncorrected errors potentially leading to a firm's insolvency and acquisition within months.
- Operational expectations include system shutdowns and restarts aligned with the trading day, significant volume spikes at market open and close, and a daily two-hour window where metadata downloads must complete before open.
- Critical trading events such as FOMC minutes (occurring every few months) and unplanned geopolitical or political shocks are identified as triggers for extreme volatility that can cause market data to go "haywire."
- Monitoring strategy rejects SLO-based metrics like error rates below 0.01% in favor of event-based alerting, which requires enumerating every edge case to ensure 100% accuracy on specific orders, despite the increased workload and potential for noisy alerts.
- The firm employs a "defense in depth" approach with redundant, uncorrelated alerts across different systems and teams, avoiding shared underlying metadata, while maintaining orthogonal monitoring sets where technical staff track system health and trading staff track trading health.
- Effective incident response relies on deep cross-functional context, allowing traders to use business-specific terminology (e.g., "spoo market data") that technical staff can instantly map to order engines without lookup tables.
- A sample incident based on real amalgamations is described where a price serialization refactor caused "fill too good" alerts at 9:31 AM, leading to an automatic order engine halt at 9:32 AM.
- During the incident, traders reporting stale market data from the 9:30 AM open revealed a missed exchange-driven change (EDC), forcing the decision to keep specific order engines halted while others continued until a configuration fix was applied.
- The incident resolution was deemed successful due to rapid reaction times of approximately one minute between actions, though post-mortems question if pre-open alerts could have prevented the open impact.
- Alerting conventions are decentralized with no top-down edicts, though common libraries exist; teams are expected to define their own alerting logic based on specific failure modes, which is labor-intensive but results in robust, reliable systems.
- Risk management utilizes a hybrid of automated system limits that halt trading upon threshold breaches and human decision-making for firm-wide risks, acknowledging that cascading alerts are not always raised if systems appear functional while trading logic fails.
- Future improvements may include using LLMs to parse exchange communications, addressing historical instances where human interpretation of such emails led to incorrect risk assessments.
- The speaker notes that trading firms do not rely on the financial system to prevent catastrophic errors, as competitors immediately exploit bad trades, making internal monitoring systems among the most robust and redundant infrastructure at the firm.