Interview, Fireside Chat, Webinar
Clock Synchronization with Chris Perl
- Clock Drift Mechanism: Computer clocks rely on internal oscillators (crystals) that drift due to environmental factors like heat; high-intensity tasks (e.g., compiling) alter the heat profile and oscillator frequency.
- Drift Magnitude: Standard modern servers typically drift between 50 to 100 microseconds per second, accumulating roughly 2 seconds of error per day without correction.
- Synchronization Definition: True synchronization requires two conditions: clocks must agree with each other within a defined error margin, and they must align with an external reference (UTC) rather than just each other.
- External Time Sources: Accurate time originates from atomic clocks (e.g., hydrogen masers) aggregated by physics labs (e.g., NPL), distributed via the GPS constellation, and received by appliances with ~100 nanosecond accuracy.
- Regulatory Driver: Jane Street redesigned its clock system to comply with European financial regulations (MIFID 2), requiring clocks to remain within 100 microseconds of UTC.
- NTP Limitations: Standard NTP implementations often fail to meet the 100-microsecond requirement because default poll intervals (minimum 8 seconds) are too coarse and software timestamping introduces significant kernel scheduling delays.
- PTP Accuracy Drivers: Precision Time Protocol (PTP) achieves microsecond accuracy through three primary mechanisms: hardware timestamping at the network interface card (NIC), higher update frequencies (default 1-second interval), and switch involvement to subtract network latency.
- PTP Reliability Concerns: Chris Pearl noted PTP's "single point of failure" vulnerability; if a GPS appliance (Grandmaster) malfunctions, PTP lacks the robust "false ticker" filtering logic present in NTP to detect and ignore bad data sources.
- Jane Street Hybrid Architecture: The solution involves a two-tiered system: GPS appliances use PTP to synchronize Linux time servers (leveraging hardware timestamping), which then act as NTP servers for the broader network using "Interleaved Mode" to transmit hardware timestamps.
- NTP Interleaved Mode: This mode allows an NTP client to include the hardware timestamp from a previous transmission in its current request, enabling precise calculation of network delay without requiring the server to support hardware timestamping on transmission.
- Achieved Accuracy: The redesigned system achieves a worst-case error of approximately 35 microseconds and a best-case error of 20 microseconds for machines with client-side hardware timestamping.
- Fault Tolerance Strategy: Clients are configured to query local NTP servers first to minimize network latency, while also querying distant servers specifically to cross-verify data and detect "crazy" timestamps from a compromised local source.
- Monitoring Protocol: Compliance is surveilled by sampling error statistics every 10 seconds and aggregating them into a database, assuming non-compliance states persist long enough to be captured at this interval.
- Future Precision Path: Achieving the next order of magnitude (10 microseconds) would likely require PTP extensions like "White Rabbit" (synchronous Ethernet + PTP) to achieve sub-nanosecond syntonization (frequency alignment), though this necessitates a robust redundancy strategy.
- Design Philosophy: The team opted to keep the network topology simple (avoiding dedicated time networks or switch-dependent configurations) to minimize operational complexity and increase inspectability, leveraging high-quality hardware rather than network fabric.