Conference Presentation
System Jitter and Where to Find It: A Whack-a-Mole Experience
- Processing data with staleness of milliseconds or 10 microseconds is expected to result in poor trading outcomes.
- Jane Street plans to maintain a focus on OCaml, utilizing a single-threaded runtime with async scheduling for blocking calls, while reserving multi-threaded execution to separate processes with potential memory sharing.
- Reducing jitter and noise is predicted to make logarithmic scale plots more interpretable than linear ones.
- High jitter on virtualized systems may stem from noisy neighbor resource contention or virtual machine overhead, with plans to eliminate this by migrating from virtual machines to bare metal hardware.
- Even on bare metal, residual noise around one microsecond is anticipated, with recommendations to use Intel processor trace for control flow analysis if standard profilers fail to identify tail latencies.
- Events exceeding a 300-nanosecond round trip are expected to trigger spikes in the "magic trace" tool, specifically indicating
ixgbe pollactivity. - Network interrupts are planned to be isolated from application cores using the
isolCPUskernel parameter andtask set, shifting interrupt handling to CPU zero to improve latencies. - Disabling timer ticks via the
nohertzfullparameter is expected to reduce kernel check frequency from approximately 1,000 times per second to once per second, further lowering latencies. - Remaining latency tails are hypothesized to result from micro-architectural issues, specifically CPU frequency rescaling (P-state transitions) causing turbo boost oscillations.
- Disabling turbo boost is predicted to remove jitter and sawtooth patterns, resulting in a stable 3.6 gigahertz base frequency, though at a slight speed reduction compared to the 4.4 gigahertz turbo frequency, a difference expected to widen with more complex workloads.
- The gap between memory latency and CPU speed is expected to worsen as chips gain more cores, necessitating speculative execution which risks pipeline stalls if bad speculation occurs.
- Adding the x86
pauseinstruction to spin loops is expected to reduce the bad speculation bucket from 8% to 1%, causing three latency distribution peaks to collapse into one. - After implementing the
pauseinstruction, the system is projected to achieve 163 nanoseconds latency with very low jitter. - Overclocking hardware with specialized memory is speculated to improve numbers but carries a risk of memory corruption and system failure if pushed too far, creating instability and trade-offs.
- Cloud environments are expected to exhibit inherent jitter, with lower-tier providers frequently oversubscribing instances.
- Running Docker on bare metal is expected to introduce moderately slower kernel interactions due to cgroup accounting but will not increase jitter absent system calls.
- Continuous performance measurement is planned to ensure regressions are detected immediately when p99 latency deviates from a low-variance median, a state preferred over high-tail systems where 10x worsening might go unnoticed.