newsfilter.io

Build Systems with Andrey Mokhov

Jane StreetAndrey Mokhov, Ron Minsky
Watch on YouTube →
  • Build systems are expected to evolve into standard compiler components, though current fragmentation by language community may persist or worsen if this integration occurs.
  • Jane Street plans a staged migration from Jenga to Dune, requiring the latter to strictly exceed the former in features; this transition involves splitting Dune into a backend engine and frontend rule description to allow frontend swapping, though it creates a period of dual-maintenance complexity.
  • The Jane Street monorepo contains approximately 15 to 20 million lines of code with full builds taking roughly one hour, while Google-scale systems with billions of lines would find full rebuilds computationally prohibitive.
  • Moving toward finer-grained incrementality (e.g., line or function level) necessitates abandoning standard command-invocation models in favor of shared memory, persistent build servers, or different operational models.
  • The Glasgow Haskell Compiler (GHC) migration from Make to Hadrian is an ongoing project five years in the future, described as an "archaeology expedition" to extract 25 years of accumulated knowledge rather than a simple code rewrite.
  • Scientific computation workflows differ from build systems by requiring orchestration across multiple locations, long runtimes (months), and upfront error detection, whereas build systems optimize for fast feedback loops.
  • The "Build Systems a la Carte" framework identifies schedulers and rebuilders as core components, allowing combinations such as Cloud Shake (Shake scheduler + Bazel rebuilder) to explore new system designs.
  • Bazel's Skylark language is deliberately restricted to prevent non-determinism and recursion to ensure safety and efficient scheduling, despite user demand for features like while loops.
  • Traditional systems like Make rely entirely on file system timestamps for state, which is often overly conservative (triggering unnecessary rebuilds) or under-conservative (failing to rebuild when timestamps are corrupted).
  • Dune was initially adopted by the external OCaml community for being 5 to 10 times faster than competitors for scratch builds, despite lacking incremental rebuild capabilities at its inception.
  • Future integration trends include closer ties with IDEs for auto-completion based on partially compiled code, as well as convergence with continuous integration and documentation infrastructure.
  • Incremental computation in financial sectors (e.g., Excel) handles massive datasets and circular dependencies, contrasting with build systems that typically avoid such convergence mechanisms.
  • Dune supports versioned build languages allowing external projects to develop independently, whereas Jenga requires a non-versioned monorepo approach where all projects share the same configuration language version.
  • The complexity of migrating legacy systems is exacerbated by the "long tail" of special conditional features and the difficulty of managing large-scale knowledge bases accumulated over decades.
  • Microsoft has released open-source tools like Build Excel and AnyBuild attempting to incrementalize build description parsing, and Hack demonstrates 10-millisecond compilation times for dynamic languages by tracking function-level dependencies.