Interview, Fireside Chat
Performance Engineering on Hard Mode with Andrew Hunter
- Hyperscaler environments are viewed as optimization "easy mode" where the massive scale transforms minor data center tax reductions of 10-20% of cycles into significant returns, contrasting with trading environments where systems intentionally idle 95-99% of the time to prioritize low latency and burst handling over high CPU utilization.
- Future system architectures are expected to require optimization across multiple timescales, ranging from sub-100 nanosecond trades and 5-microsecond tails to reducing 50-millisecond tails to one millisecond, with hardware acceleration via FPGAs targeted to achieve turn-around times under 100 nanoseconds, an order of magnitude faster than software.
- Hardware acceleration is limited to "hyper-focused speed things" and is considered "flatly impossible" for complicated logic, necessitating a software architecture that handles complex decision-making while offloading simple, single-purpose speed tasks to hardware, which also faces a 24-hour compiler runtime in Verilog and PCI Express latency bottlenecks preventing standard systems from breaking the 1-microsecond packet processing barrier.
- The C compiler ecosystem is expected to remain 30 years ahead of the OCaml compiler, making catching up on code generation unlikely, though interoperability with C libraries is projected to incur only 3-4 nanoseconds of overhead compared to 300-400 nanoseconds in Java.
- Performance engineering efforts are expected to prioritize "mechanical sympathy" and architectural fixes over micro-optimizations, with the most impactful gains potentially coming from optimizing business logic rather than infrastructure, while future compiler changes aim to provide greater control over OCaml type layouts and improved visualizations like "magic traces" for analyzing 99th percentile tails.
- Rapid iteration on trading strategies is expected to shorten feedback loops to approximately 10 minutes, contrasting with historical processes like bourbon aging that require 5 to 10 years per iteration, supported by tooling developments that move away from esoteric knowledge toward usable defaults and "idiot-proof APIs."
- Hardware constraints such as Intel's integrated Processor Trace are expected to carry a material performance penalty of 5-15% when active, requiring selective enabling rather than fleet-wide deployment, while memory allocation traces are anticipated to offer more value than statistical profilers by providing literal traces of allocation lifecycles to identify convergence points.
- The difficulty of performance engineering is expected to lie in the weeks of investigation required to prove hypotheses rather than the implementation of resulting changes, with a noted risk that focusing on low-priority slow code could divert discipline from more critical architectural improvements.
- Historical research simulations are described as allowing "easy mode tactics" similar to hyperscalers, where throughput is king and users can accept result times of an hour, a day, or a week, provided they can process large volumes of input data.
- Zero Alloc OCaml optimizations are expected to require selective application due to the increased development difficulty, which may reduce the likelihood of developers performing necessary tasks correctly and inadvertently incur performance costs if other priorities take precedence.
- The competitive landscape is expected to see Intel offering a huge advantage through silicon-integrated features that are impossible to implement in the kernel, despite a lack of investment in open-source tooling, while AMD is not identified as having comparable advantages.