Effective ML

Speaker Profile & Context

• Yaron Minsky is a programmer at Jane Street Capital, a proprietary trading firm specializing in electronic market making.
• His background is in distributed systems (Cornell graduate work) rather than programming languages; he was converted to ML by peers who were already converted.
• Jane Street handles billions of dollars in nominal trading volume daily, requiring extreme focus on correctness to mitigate financial risk from software errors.
• The firm's development culture is "code review driven," prioritizing human readability and deep understanding over automated testing alone to catch rare, tail-risk events.

Core Philosophy: Reader Over Writer

• Code is fundamentally read and modified many more times than it is written; therefore, the interests of the reader must always take precedence over the convenience of the writer.
• Prioritizing the reader's perspective ensures code remains simple, clear, and capable of evolving correctly over time.
• This philosophy drives specific design choices such as writing interfaces, avoiding boilerplate, and favoring explicit patterns over "clever" abstractions.

Interface & Standardization

• Uniform Interfaces: Teams should define and enforce standard module signatures (e.g., comparable) using include statements in .mli files to ensure consistency across the codebase.
• Enforcement: Standardizing interfaces via compiler-enforced constraints is more effective than cultural reminders, as it prevents deviations before compilation.
• Functor Compatibility: Uniform interfaces allow functors (functions from module to module) to be applied broadly without requiring ad-hoc refactoring or "packing" of modules to fit specific signatures.

Data Modeling & Invariants

• Illegal States Must Be Unrepresentable: Designers should leverage Algebraic Data Types (ADTs) to structure types such that logically impossible states cannot be constructed by the type system.
• Refactoring Example: A connection state management example was refactored from a record with option fields (allowing inconsistent states) to a sum type where each variant (e.g., connecting, connected) contains only the fields relevant to that specific state.
• Tuple Grouping: Related fields (e.g., ping_time and ping_id) should be grouped into tuples within variants to enforce that they always exist together, removing the possibility of partial data.

Exhaustiveness Checking

• Compiler Enforcement: Match expressions must be exhaustively checked; the compiler should be configured to refuse compilation if a pattern match does not cover all cases.
• Refactoring Safety: Exhaustiveness checks act as a critical safety net during refactoring; if a new case is added to a variant type, the compiler will immediately flag all missing pattern matches.
• Risk of Catch-All: Using wildcard patterns (_) to ignore new cases allows bugs to persist silently when the underlying data model evolves, as the code will not fail to compile even if logic is incomplete.

Naming & Scope Management

• Avoiding open: Excessive open directives should be avoided as they obscure the origin of identifiers, making code difficult to read and debug.
• Local Renaming: When brevity is required, use local let bindings to rename long module paths (e.g., let f = command.flag) rather than globally opening modules.
• Cognitive Load: Local bindings ensure that readers only need to remember short-term definitions within a specific scope, preventing cognitive overload.

Error Handling Conventions

• Explicit Naming: Functions that may raise exceptions must have distinct naming conventions (e.g., mem for safe returns, mem_exn for exceptions) to make error behavior immediately apparent.
• Contextual Errors: Whether a condition is "exceptional" depends on context; explicit naming allows the same logical operation to be safe in one context and error-raising in another.
• Avoiding Boilerplate: Repeated code patterns (boilerplate) should be abstracted into local functions to reduce the likelihood of errors during updates and to prevent readers from skimming over dull, repetitive sections.
• Abstract vs. Understandable: Abstraction should not be used if it increases complexity; overly clever type-level tricks or lambda calculus abstractions that obscure logic are detrimental to maintenance.

Side Effects & Purity

• Balanced Approach: While purity aids reasoning, side effects are sometimes necessary for performance and natural modeling; the goal is to segment side effects clearly rather than avoid them entirely.
• Performance Constraints: High-frequency trading systems require performance optimizations that may necessitate mutable state, which should be accepted when it results in simpler, more efficient code.
• Complexity Enemy: The primary enemy of correctness is complexity, not the presence of side effects; if purity forces complex code, a side-effecting approach is often superior.

Advanced Type Techniques

• Phantom Types: Techniques like phantom types allow the type system to enforce invariants (e.g., read-only vs. read-write handles) without impacting runtime performance, as the type tags are erased at compile time.
• Capability Control: Phantom types enable "capability-style" access control, where a single underlying memory location can have multiple handles with different permissions enforced strictly by the type checker.
• Polymorphism: Unlike creating distinct types for each capability, phantom types allow for polymorphic code that operates on both read and write handles simultaneously, with the compiler inferring the correct specific type based on usage context.

Industry Constraints & Reality

• Jane Street Codebase: The firm maintains a codebase of 1–2 million lines in OCaml, described as potentially the largest ML codebase in the world.
• Library Limitations: OCaml lacks mature libraries for web UIs and GUIs, often forcing the firm to use legacy technologies like curses for interfaces.
• External Integration: The firm prefers bespoke software development over integrating external proprietary databases or UI toolkits, allowing full control over the codebase.
• C Interop: While the vast majority of code is written in OCaml, dropping down to C is sometimes necessary for specific low-level operations or performance-critical sections.
• Generic Serialization: Implementing generic printers or serializers for custom types can be painful in ML, often requiring macro systems or advanced type-level programming to solve effectively.