Swapping the Engine Out of a Moving Race Car with Ella Ehrlich

System Overview

• Gord is a legacy Jane Street system (15 years old) that ingests, normalizes, and distributes global trading data.
• Core functions include calculating real-time positions, uploading trades to clearing firms, settling trades, and ensuring regulatory compliance regarding short sales.
• The system maintains a strict uptime guarantee because any outage halts all firm trading; it is designed to fail in constrained, isolated ways (e.g., per-exchange failure) to prevent cascading catastrophic outages.
• Architecture includes a primary production instance and a standby instance in a separate data center to allow for graceful failover during network partitions or hardware failures.
• Performance scaling targets a 2x–5x multiplier over the busiest historical trading days, as peak loads tend to exponentially exceed previous records.

Legacy Design Challenges & Evolution

• Data Normalization: Originally performed centrally, the normalization of symbology (e.g., converting exchange-specific codes to internal standards) and user identities is shifting to the source systems where the data originates to reduce the load on Gord.
• Synthetic Data Generation: The system was designed to "make up" missing orders when fills arrived without corresponding orders; this is now recognized as a flawed design pattern the team is actively working to unwind.
• Schema Rigidity: The original protocol used a flexible "string-string map" (bag of fields) to avoid versioning issues, but this made adding new transaction types (e.g., first-class "reject" types) impossible without breaking downstream clients.
• Migration to Kafka: To solve scaling bottlenecks and the inefficiency of push-based TCP connections, Gord migrated its distribution layer to Kafka.
  • This shift changed the model from push-based (where slow clients caused memory buffering and disconnection) to pull-based (where clients fetch data at their own pace).
  • The migration eliminated per-client snapshot generation costs by utilizing Kafka's disk-based storage and efficient offset management.
• Standardized Input Library: A new internal library is being developed to enforce required fields and data types for upstream sources (using FIX protocol), replacing informal agreements and reducing the "guessing" game for data providers.

Operational Improvements & Process Changes

• Testing Frameworks: The team transitioned from manual "diff checking" in development environments to automated in-unit tests and performance benchmarks.
• Release Cadence: Deployment frequency increased from multiple times per year to approximately once per month due to improved test coverage and automation.
• Rollout Automation: The manual, time-sensitive symlink rotation process across global offices (NY, London, Hong Kong) was replaced by a config-file-driven automation system to eliminate human error risks during global day transitions.
• Stream Segmentation: The system evolved from offering only "full stream" or "fills only" views to supporting "monitoring" and sub-streams (e.g., options-specific data) to reduce client-side filtering overhead.
• Future Goals: The long-term engineering objective is to achieve 24/7 availability without daily shutdowns, moving away from the historical "trading day" reset model.

Diversity & Recruiting Initiatives

• Women in STEM & Insight: Jane Street runs outreach programs targeting high school and college women; the "Insight" program (a one-week intensive for sophomores) has a ~20% conversion rate to future internships.
• Recruiting Outcomes: Female representation in the dev intern class has grown to 25% (up from near 0% a decade ago), though the firm notes lateral hiring of women remains a challenge.
• InFocus Program: A new educational program for underrepresented minorities (freshman/sophomore year) focuses on teaching OCaml and open-source contributions to level the playing field for candidates lacking prior industry exposure.
• Community Building: The firm hosts casual "brunches" and community events to build reputation and trust, emphasizing that genuine relationships are a primary driver for candidate interest.
• Strategic Philosophy: Programs are designed to be "legitimately helpful" (educational value) regardless of immediate hiring intent to foster long-term brand loyalty and broaden the candidate pipeline.