Lecture, Conference Presentation, Keynote
Safe at Any Speed: Building a Performant, Safe, Maintainable Packet Processor
- Jane Street anticipates that sub-microsecond processing (750 nanoseconds or less) is the threshold for avoiding queue growth and latency issues, whereas exceeding one microsecond risks system failure to keep up with market data, causing buffers to fill to full volume at market close and resulting in severe latency spikes for post-peak messages.
- To maintain performance, the firm expects single-core packet processing to prioritize speed above all else, utilizing "zero-allocation" paradigms such as flat representations, avoiding high-level abstractions like async or monads, and implementing custom immediate option types or PPX-generated code to eliminate per-message allocations (e.g., saving four bytes per
optionvalue). - The outlook includes a prediction that naive, eager parsing strategies allocate records for every message, creating a performance gap where optimized approaches are roughly eight times faster (700 nanoseconds versus five microseconds) due to improved cache locality.
- Plans involve architectural strategies to isolate exchange-specific protocol irregularities at system edges for code reuse and to access memory outside the garbage collector via C Foreign Function Interface for explicit memory management on critical paths.
- Jane Street expects the OCaml runtime to pose specific constraints, including "stop the world" garbage collection that precludes parallelism, immediate values stored on registers, and the risk that excessive allocation causes cache misses affecting median performance rather than just tails.
- The summary includes expectations that function calls in tight loops may cause closure allocations or instruction cache misses, and while compiler inlining can mitigate this, aggressive inlining requires empirical heuristics based on function size or code path tagging rather than theoretical models.
- Jane Street predicts that operating system stack usage can dominate execution time even in fast applications, necessitating techniques such as zero-copy, kernel bypass, or FPGA hardware for specialized scenarios below standard software speeds.
- The outlook notes that performance optimizations are fragile and may break with compiler or runtime version changes, requiring constant measurement and regression benchmarks to ensure stability.
- Expectations cover the "cold hardware tax," where minimal workloads perform poorly compared to those doing more work, and the need for developers to reason about avoiding these states while balancing the maintenance costs of complex code generators against early, precise performance analysis.
- Plans include maintaining a single-language workflow with OCaml to leverage high-value tooling and maintainability, even when considering alternatives without garbage collectors, and ensuring synchronous processing to manage mutable state safely.
- The firm expects that high-level languages like OCaml can achieve sub-microsecond goals without assembly language, provided runtime behavior and machine code translation are deeply understood, though FPGAs remain reserved for highly specialized applications requiring speeds unachievable by software.