Anil Madhavapeddy on What is an Operating System

Anil Madhavapeddy — Professor of Computer Science at Cambridge and the creator of MirageOS — joins host Ron Minsky for a wide-ranging conversation about what an operating system actually is, why the original multiuser vision quietly failed, how unikernels emerged as a radical alternative, and what the coming multicore OCaml runtime means for systems programmers.

Key ideas

1. An operating system is a mediator between software and hardware — and that role has been steadily hollowed out. The kernel (the core of an OS — the always-on layer that manages memory, input/output, and scheduling on behalf of every other program) was designed to let many users share one expensive machine. Virtualisation and containers have since made each application its own tenant, reducing the kernel to a compatibility layer rather than a genuine shared resource.
2. MirageOS treats the kernel as just another library. Instead of compiling your code into a binary that then runs atop Linux, the MirageOS compiler continues past that point and emits a unikernel — a self-contained, bootable image containing only the operating-system components the application actually uses. The result is vastly smaller attack surface, sub-second reboots, and a binary that can run on bare metal, inside a hypervisor, or on a microcontroller without changing the source code.
3. The Xen hypervisor showed that hardware is the best abstraction. Xen (the virtualisation layer beneath most cloud computing) works by paravirtualisation — fooling a guest operating system into thinking it runs on real hardware whilst a thin hypervisor shim handles the real mapping. This allowed entire OS stacks to be packed onto one physical machine with minimal performance cost, redefining the ‘user’ as someone who boots an operating system rather than logs into one.
4. OCaml’s module system makes evolving interfaces tractable. MirageOS defines every device or protocol as a module signature (a formal contract saying what operations exist and what types they consume and produce). Concrete libraries implement those signatures; multiple implementations — a Linux socket, a pure-OCaml TCP stack — can satisfy the same signature interchangeably. When hardware or security requirements change, only the relevant signatures and implementations need updating.
5. OCaml’s coming multicore support is designed to be the least surprising parallel runtime of any mainstream language. The multicore team proved that data races in OCaml are bounded in space and time — unlike C++ (where a race is undefined behaviour that can corrupt the rest of the program) or Java (where a race can produce incorrect results across future executions). The programmer’s experience will be ordinary parallel libraries (‘domainslib’), fast, with no hidden traps.

Content

What is an operating system, and why has it failed?

Ron Minsky opens with a provocation: the original operating system was a multiuser sharing mechanism, designed when computers cost as much as buildings. Unix, Multics, and their successors built elaborate abstractions — processes, file permissions, isolation — to let many people safely share one machine. Then, gradually and almost invisibly, that premise evaporated. Virtualisation (running multiple OS instances on one machine), then containers (packaging an application with just enough OS to run it), each represent a further retreat from the ideal of genuine sharing towards the pragmatic goal of reproducible, isolated deployment. Madhavapeddy accepts the framing: what we call an operating system today is less a governor of shared resources than a compatibility shim frozen in place by decades of software written against its interfaces.

MirageOS: the compiler that refuses to stop

Madhavapeddy’s PhD observation was simple. He had rewritten several internet protocols — DNS (the service that translates human-readable names like google.com into network addresses), SSH (the protocol computers use to talk to each other over remote connections) — in pure OCaml, proving they could match C for speed whilst being safer and formally analysable. But at the end of the compilation process, the OCaml compiler stopped, emitting a binary that then depended on 25 million lines of Linux C code for every file-system access, network packet, and timer interrupt. Why drag all of that along?

MirageOS answers by extending the compilation process. Rather than generating a binary that calls into the OS, it generates an OS. The compiler evaluates application logic and configuration together — both written in OCaml — and emits a unikernel: a complete, bootable image containing only the networking, file-system, and driver libraries the application actually uses, all written in pure OCaml. A unikernel reboots in twenty milliseconds, presents a minimal attack surface (there is no shell, no unused driver, no unneeded system call), and can be cross-compiled from a full Linux environment to a tiny ESP32 microcontroller by incrementally narrowing the interface specifications.

This lineage traces back to the Xen hypervisor, which itself began as a bet in a Cambridge pub. Keir Fraser hacked a working prototype over a weekend; a team including Madhavapeddy productised it at XenSource. Xen’s paravirtualisation technique — shimming a thin layer between guest OS and real hardware so that multiple OS instances share one machine — became the foundation of cloud computing. MirageOS was originally the minimal OCaml test harness for Xen’s lowest-level interfaces. The TCP/IP stack came next, then HTTP; the unikernel concept emerged organically from the accumulation.

The library operating system model

In a conventional OS, the kernel (think: the privileged core that owns the hardware) is a monolith that every user-space process asks for resources via system calls — requests that cross from the unprivileged application layer into the privileged kernel layer. Library operating systems invert this: the kernel is just another library linked into the application, no different in principle from a graphics or cryptography library. The trade-off is multiuser isolation — only one application can hold exclusive access to the hardware at once. For specialised, single-purpose deployments (a web server, a VPN gateway, a climate sensor) this is acceptable; for a general-purpose desktop it is not.

Madhavapeddy illustrates with a concrete experiment: a solar-powered website on Raspberry Pis around the world. The team first wrote a normal OCaml Unix web server, measured its energy use, found it exceeded the solar budget, then progressively narrowed the MirageOS interface — removing unused syscalls and file-system features — until the application could run on an ESP32 microcontroller, cutting energy consumption dramatically without rewriting any application logic.

OCaml modules and the art of evolving interfaces

The reason MirageOS can evolve without breaking everything is OCaml’s module system. A module signature (analogous to a contract in law — it specifies what obligations a party must discharge without prescribing how) defines what operations a networking stack, file system, or hardware driver must expose. Any library that satisfies the signature is a valid implementation. MirageOS has hundreds of signatures and hundreds of concrete implementations; swapping the underlying technology (say, moving from a Linux socket to a pure-OCaml TCP stack, or from a POSIX file system to a custom distributed one) requires only that the new library satisfies the existing signature.

This design also handles multilanguage systems. The Tezos blockchain node, for example, combines OCaml and Rust — Rust at the lowest runtime layers (where its ownership model — a compiler-enforced rule that each value has exactly one owner, preventing accidental aliasing — is well suited), OCaml higher up. Docker for Mac and Docker for Windows route every container file-system and network operation through a MirageOS translation layer that converts Linux container semantics into native macOS or Windows calls. Support incidents dropped by roughly 99% after deployment.

The Bitcoin piñata and the security argument

In 2015, the Robur team wrote a complete TLS stack (the cryptographic protocol that secures HTTPS) in pure OCaml after the Heartbleed vulnerability exposed a critical flaw in OpenSSL’s C implementation. To demonstrate the unikernel’s security properties, they hid ten bitcoins inside a unikernel, published the private keys, put it on the public internet, and invited the world to take the money. Hundreds of thousands of attacks followed; denial-of-service attempts knocked the unikernel offline, but it rebooted in twenty milliseconds and came straight back. Nobody took the Bitcoin.

Multicore OCaml: six years of research landing in two modules

OCaml has historically restricted parallel execution on a shared heap — a restriction that has long frustrated systems programmers. The multicore project, led by KC Sivaramakrishnan working with Madhavapeddy, spent six years solving three interlocking problems: a memory model (what happens when two threads read and write the same value simultaneously), a multicore garbage collector (reclaiming memory safely even as multiple cores allocate and die concurrently), and backwards compatibility (existing OCaml code compiled with OCaml 5.0 should run no slower than before).

The memory model work (published at PLDI) produced a result called LDRF — local data-race freedom — which guarantees that a data race (two threads accessing the same memory without synchronisation) cannot spread its damage through the program. In C++, a data race triggers undefined behaviour for the entire remainder of execution; in Java, the consequences can propagate across future executions of the racing code. In OCaml 5.0, a race is localised: its effects are contained to the immediate context, so debugging is tractable and the rest of the program remains well-defined. The performance overhead is unmeasurable on x86 and under 2% on ARM and PowerPC.

The programmer-facing interface is deliberately minimal: two new modules that expose domains (think: one per physical CPU core) and, in a later release, fibers — lightweight micro-threads, of which a program might run millions, migratable between cores, with scheduling logic written in ordinary OCaml rather than hardwired into the kernel. The effect system that enables fibers also powers EIO, a new I/O library that uses native OS facilities (io_uring on Linux, Grand Central Dispatch on macOS, IOCP on Windows) transparently, so the programmer writes straight-line code and the runtime handles concurrency automatically.

A career that started with a crashed Mars lander

Madhavapeddy came to computer science obliquely. He trained as an electrical engineer, stumbled into a NASA internship via a programming language he encountered in an online game, and set up the infrastructure that broadcast the Mars Polar Lander’s descent to one in three people on the internet in 1999. The servers were hacked within hours of going live; he replaced them with OpenBSD (a Unix variant designed for reliability and security), sent in patches when he found bugs, and received the maintainers’ approval — an early dopamine hit that set the course. When the lander crashed into Mars at high speed and the infrastructure he had built was never needed, he returned to Cambridge to do a PhD and has worked at the intersection of operating systems, functional programming, and open-source community building ever since.

OCaml Labs, which he co-founded with Ron Minsky, provided the institutional glue for the OCaml ecosystem — accepting maintainership of tools like Merlin and OCaml Format, coordinating the Dune build system with Jane Street, and running the formal processes that allowed academic researchers to graduate into industrial contributors. Stephen Dolan and Leo White are among the alumni. David Allsopp, one of the most prolific contributors to the core compiler, was also maintained his career as a countertenor singer throughout — a pairing that required some creativity to explain to Cambridge HR.

Related

• Anil Madhavapeddy — speaker
• Ron Minsky — host
• Stephen Dolan on Memory Management, Garbage Collection, and OCaml — companion episode on OCaml’s garbage collector and memory model, with Stephen Dolan who is also mentioned by Madhavapeddy as an OCaml Labs alumnus
• The Future of Software Engineering — theme; MirageOS and unikernels represent one answer to the question of what systems programming looks like when safety is the default