Conference Presentation
Caveat Configurator: How to replace configs with code, and why you might not want to
- Jane Street, a proprietary trading firm of ~500 people (150 developers), operates under two primary constraints: maximizing engineer output flexibility and ensuring absolute system safety.
- The firm trades across ~90 countries with a high ratio of systems per engineer, necessitating systems that can be instantiated quickly with minor tweaks to avoid costly development cycles.
- A "big bug" in a trading system represents an existential risk to the firm, whereas smaller bugs result in missed opportunities or downtime.
- In 2014, Jane Street adopted "OCaml Plugin," a library enabling dynamic loading of OCaml code to create a flexible plugin architecture for configuration.
- The initial deployment of OCaml Plugin in production (e.g., a new email system) resulted in a "flaming disaster" due to naive implementation of the plugin architecture.
- Configuration strategies discussed range from "zero-config" (hardcoded defaults requiring redeployment) to "pure data" (key-value pairs, JSON, YAML) to "custom DSLs" (Domain Specific Languages).
- Custom DSLs were rejected due to the difficulty of language design, lack of existing tooling (IDE support, testing frameworks), and high maintenance burdens on developers.
- The "code-based configuration" approach allows for arbitrary expressiveness but introduces significant risks when untrusted code runs inside the production process.
- A critical failure of the naive plugin approach was the lack of versioning discipline; when core application dependencies changed, plugins often broke because they were compiled against outdated library versions.
- Unplanned plugin usage led to the abandonment of applications because the scattered nature of plugin files and lack of centralization made deployment and version management unviable.
- Tooling friction persisted despite using a programming language; standard IDE features (jump-to-definition, auto-completion) often failed because plugins existed outside the main codebase.
- A specific production email server configuration grew to 4,000 lines of OCaml plugins, highlighting the lack of transparency and manageability compared to the original 13-line pure data config.
- Jane Street initially resolved these issues by bundling plugins with the core application, effectively removing runtime dynamism in favor of a deployable, version-controlled unit.
- Two current strategies emerged: "Config Generation" and "PlugD" (Centralized Plugin Infrastructure).
- Config Generation involves writing OCaml scripts to generate static, versioned configuration files (pure data), allowing the use of code for complexity while retaining the safety and tooling benefits of flat data.
- With Config Generation, generated diffs provide clear visibility into configuration changes, and the process is integrated into standard code review and build workflows.
- PlugD is a centralized system built to safely manage runtime-dynamic plugins by treating them as first-class code with mandatory code review, testing, and compilation.
- PlugD decouples the application from plugin file locations and manages library versioning by serving only vetted, compatible plugin versions.
- A decision framework exists: if true runtime dynamism is not needed, use Config Generation; if plugins are needed but must be decoupled from the app, use PlugD; otherwise, bundle plugins with the app.
- A key insight is that plugins were often used to enforce a mental separation between core and configurable logic, which could often be achieved through better application structure without the overhead of plugins.
- Traders at Jane Street frequently write OCaml plugins, necessitating a system where application developers can review and validate trader-written logic to ensure safety.
- The "versioning problem" stems from runtime dynamic linking requiring the application to bundle compiler artifacts; if a plugin uses a different library version than the bundled artifacts, compilation at runtime fails.
- Jane Street utilizes a library-based build model where applications are constructed by composing libraries, blurring the line between "configuration" and "application instantiation."
- Plugin sandboxing is technically possible via worker processes to isolate crashes but is not the dominant use case due to architectural complexity.
- The PlugD infrastructure was built by one person and is currently owned by one person ("Wong"), though knowledge is distributed to mitigate single-point-of-failure risks.
- Plugins are frequently migrated into the core application in future versions when a feature becomes stable and widely used enough to warrant integration.