Andrey Mokhov on Build Systems

Andrey Mokhov — software engineer at Jane Street and co-author of the paper Build Systems à la Carte — joins host Ron Minsky for a deep examination of how build systems work, why so many of them exist, and where the field is heading as the boundaries between build tools, compilers, and incremental computation engines begin to dissolve.

Key ideas

1. A build system is an incrementality engine: it figures out which tasks need re-running and skips the rest. The simplest approach — a shell script — reruns everything every time. Make, invented in 1976, was the breakthrough: you declare each task’s inputs, outputs, and the command to produce them, and Make only reruns tasks whose inputs have changed since their outputs were last produced. That one idea — running less work by tracking what changed — defines the core service every build system must provide.
2. Make uses file timestamps to detect change, which is both too conservative and not conservative enough. Add a comment to a source file, recompile it, and Make will trigger recompilation of everything downstream — even though the compiled output did not change. This failure mode, called the absence of early cutoff, causes unnecessary rebuilds. Going the other way, backup software or other tools that alter modification times can fool Make into skipping a needed recompile. Both problems stem from the same root: timestamps are a proxy for content change, not the real thing.
3. ‘Build Systems à la Carte’ untangles the ecosystem by decomposing every build system into a scheduler and a rebuilder. The scheduler (think: task manager) decides the order in which tasks run; the rebuilder (think: cache policy) decides whether a given task needs to be re-executed. Mokhov and his co-authors catalogued four schedulers and three rebuilders and showed these could be combined freely — exposing new combinations that nobody had implemented, including Cloud Shake, a hybrid of Shake’s dynamic scheduling and Bazel’s content-hash cloud cache.
4. Cloud builds and memoisation are how modern systems scale to hundreds of developers. In Make, every developer rebuilds from scratch on their own machine. In a cloud-aware system such as Bazel, each task is identified by a hash of its inputs and its rule description; if anyone on any machine has already run that exact task, the result sits in a shared store and can be downloaded rather than recomputed. The mechanism is memoisation (caching the result of a pure computation so it need never be repeated) applied at the level of build tasks rather than individual function calls.
5. The long-term direction is build systems as a first-class component inside compilers, enabling sub-file incrementality. Today’s build systems operate at file granularity: a changed file triggers a re-task. Modern IDEs and compilers like the Hack type checker already operate at function granularity — if one function changes, only the functions that type-check against it need re-examination. Mokhov expects this to generalise: build systems will eventually expose APIs that let compilers register individual type-checking or code-generation tasks, bringing the same incrementality discipline inside the compiler itself.

Content

What a build system actually does

Ron Minsky opens by noting that most developers think about their build system only when it breaks. Mokhov’s answer to ‘what is a build system?’ is deliberately practical: it automates the tasks a developer would otherwise run by hand — compilation, testing, documentation generation — and, crucially, it works out which tasks need to be run. Writing a shell script that runs everything handles the ordering but not the incrementality. Build systems exist because projects grow, compilations take time, and running every step every time becomes intolerable.

Make’s model is simple: a build graph (think: a flowchart) where each node is a task and each edge is a dependency — ‘this file must exist before that command can run.’ Given that graph, Make runs only the tasks whose recorded output files are older than their input files. It needs no external state: the file system’s modification timestamps are the bookkeeping.

Make’s limitations: timestamps, scale, and programmability

Mokhov identifies three layers of Make’s shortcomings.

Timestamps as change proxies. Adding a comment to a source file changes its timestamp without changing the compiled output. Make therefore triggers all downstream tasks unnecessarily. The correct solution — early cutoff — requires storing extra state (hashes of task outputs), which Make deliberately avoids, since its statelessness is part of its simplicity. make clean — throwing away all recorded state and rebuilding everything — is the canonical escape hatch when timestamps lie.

Single-machine scope. In a company with hundreds of developers all working in the same codebase (a monorepo — a single version-controlled repository for all projects, which avoids the ‘dependency hell’ of matching version numbers across separate repos), every developer rebuilds everything locally even when a colleague completed the identical build minutes ago. Modern systems address this by uploading build outputs to a shared cloud store, identified by content hash. Mokhov notes that Jane Street’s internal build system, Jenga, is mid-migration to add exactly this capability.

Make as a programming language. Large build descriptions in Make use macros (rules that generate other rules) because the base language offers no abstraction mechanisms — it is a global, mutable namespace of string variables. Mokhov spent five years on Hadrian, the replacement build system for GHC (the Glasgow Haskell Compiler, a million-line codebase), and describes encountering Makefiles with lines fifty characters long containing twenty dollar signs — each a macro indirection — that nobody fully understood. Migrating such code requires archaeology more than engineering: tracing undocumented decisions made by developers who may no longer be reachable.

Build Systems à la Carte

The paper Mokhov co-authored with Neil Mitchell and Simon Peyton Jones asks a different question: instead of comparing feature lists, can one describe the algorithmic core of every major build system in twenty to thirty lines of code?

The answer is yes, using the scheduler / rebuilder decomposition. The scheduler decides the traversal order — does it restart tasks that discover new dependencies mid-run, or does it require all dependencies to be declared upfront? The rebuilder decides the cache key — is it a file timestamp (Make), a hash of the task’s inputs and rule description (Bazel’s constructive traces approach), or a record of which inputs were actually read during the last run (Shake’s verifying traces)?

Mixing these components reveals gaps. Cloud Shake — the paper’s novel combination — pairs Shake’s suspending scheduler (which pauses a task the moment it discovers an unbuilt dependency, then resumes it when that dependency finishes, handling parallelism across the suspended tasks) with Bazel’s content-hash rebuilder. Before the paper, Neil Mitchell knew he wanted to add cloud build support to Shake but had no clear blueprint. The paper showed him exactly which component to add and what its interface must look like.

Jane Street’s two build systems

Jane Street found itself maintaining two build systems through a combination of accident and organic growth.

Jenga is the internal system: written in OCaml, configured in OCaml, and tuned for monorepo builds at scale. Its user-facing language is deliberately constrained — developers declare a library’s name, dependencies, and flags, but cannot write arbitrary logic — so that the system can analyse the entire build graph statically and schedule it efficiently. Arbitrary user code would make static analysis impossible and open the door to non-deterministic builds, which in turn would break cloud caching (if the same inputs can produce different outputs, the hash-based lookup key is meaningless).

Dune began as a tool Jérémie Dimino wrote to make it easy to release Jane Street’s open-source OCaml libraries. It initially had no incrementality at all, but it was so fast at from-scratch builds, and so much easier to port than alternatives, that the broader OCaml community adopted it. Incremental builds were added later; the community contributed features; and Dune eventually grew more capabilities than Jenga while also being faster. Jane Street now plans to migrate onto Dune — a migration complicated by the same knowledge-archaeology problem Mokhov encountered on Hadrian: tens of thousands of lines of Jenga build rules encode institutional knowledge about how Jane Street’s software must be compiled, and that knowledge must be transferred, not just the code.

The current approach is to separate Dune’s front-end (the OCaml-specific rules) from its back-end (the incremental computation engine). Once separated, it becomes possible to swap one layer at a time — bridging Jenga’s front-end rules onto Dune’s back-end, for example — rather than cutting over all at once.

Build systems and the wider space of incremental computation

The Build Systems à la Carte paper includes Excel as a build system — a deliberate provocation. Excel spreadsheets are dependency graphs: a formula in one cell depends on values in others, and changing a cell should trigger only the recomputation of downstream formulas, not a recalculation of every cell. The incremental requirements are identical; the implementation details differ because Excel does not work at file granularity and explicitly supports circular dependencies with controlled iteration to a fixed point.

Minsky draws out two further extensions. Jane Street’s Incremental library constructs in-memory incremental computations in OCaml — faster timescales and lighter-weight operations than a file-level build system, but the same scheduler / rebuilder logic underneath. Scientific computation pipelines (genomics, financial simulations) resemble builds scaled to days-long jobs and multi-city data sets; the difference is that data locality matters (you do not move terabytes across cities), computations are far too long to allow a misconfiguration to surface mid-run, and the parallelism is measured in hundreds of machines rather than cores.

Mokhov sees these as points on a spectrum: fine-grained in-memory incrementality at one end, multi-week distributed scientific computation at the other, with file-level build systems in the middle. The conceptual primitives — tasks, dependencies, caching, scheduling — are the same; the engineering constraints differ widely.

IDE integration and the push toward sub-file granularity

Modern development environments increasingly demand finer granularity than a file. An IDE offering real-time auto-completion must run something like compilation on every keystroke, which requires the build system to expose information to the editor rather than merely consuming it. Minsky describes Facebook’s Hack compiler — which, on a git rebase touching thousands of files, finishes re-type-checking within ten milliseconds — as the existence proof that function-level incrementality is achievable, at the cost of a custom compiler architecture built around it.

Mokhov expects build systems to move in that direction: not just orchestrating file-level tasks but providing APIs through which a compiler can register individual type-checking jobs, get back results for the functions it has already checked, and generate further jobs based on those results. The challenge is the overhead model: at fine enough granularity, deserialising all the state needed to run a task costs more than the task itself, which points toward persistent build servers and shared-memory architectures rather than the current model of invoking a fresh process per task.

His long-run expectation: the build system will become a standard named component inside a compiler — as familiar as the parser, type checker, and code generator — rather than an afterthought bolted on to make the whole thing actually run.

Related

• Andrey Mokhov — speaker
• Ron Minsky — host
• Stephen Dolan on Memory Management, Garbage Collection, and OCaml — also Signals and Threads; both conversations concern the engineering internals of Jane Street’s OCaml toolchain
• The Future of Software Engineering — theme; the conversation’s closing section on IDE integration and compiler-level build systems speaks directly to how software development tooling is evolving