Chris Perl on Clock Synchronization

Chris Perl — systems engineer at Jane Street — joins Ron Minsky to trace how computer clocks work, why they drift, and how Jane Street rebuilt its timekeeping infrastructure to meet a European regulatory requirement that every clock stay within 100 microseconds of UTC.

Key ideas

1. Computer clocks are driven by a physical oscillator — a crystal — whose frequency wanders with temperature, and this drift is the root problem. A server-grade oscillator may walk off by 50–100 microseconds per second; over a day that compounds to several seconds of error. Getting accurate time is not just a matter of setting a clock once; it requires continuously correcting the rate at which the clock advances.
2. NTP’s core trick is four timestamps and a symmetry assumption. The client records when it sends a packet; the server records when it receives and when it replies; the client records when the reply arrives. From those four numbers you can compute the offset (how far your clock differs from the server’s) and the round-trip delay — but only if you assume the packet took equally long in each direction. Asymmetric network paths introduce systematic error that no amount of sampling can remove.
3. PTP’s extra precision comes from hardware timestamping and faster updates, not a fundamentally different calculation. PTP records timestamps at the network-interface card the instant a packet arrives, bypassing scheduling delays and interrupt latency. It also sends synchronisation messages once per second rather than once every eight seconds by default. The underlying maths is the same four-timestamp exchange; the improvement lies in making those timestamps as accurate and close-in-time as possible.
4. PTP’s robustness story is weaker than NTP’s. NTP was designed to poll multiple servers, detect falsetickers (servers reporting wrong time), and gracefully ignore them. PTP concentrates on a single grandmaster — typically a GPS appliance — and the best master clock algorithm selects for clock quality, not correctness: a misbehaving appliance that claims excellent quality will propagate bad time to the whole network indefinitely. A GPS appliance sending a wild timestamp during a maintenance shutdown illustrated the failure mode directly.
5. Jane Street’s solution layers the two protocols: PTP from GPS appliances to a tier of Linux servers, NTP from those servers onward. The Linux tier gets hardware-timestamped PTP from the GPS appliances (which support PTP hardware timestamping but not NTP hardware timestamping), then serves NTP with interleaved-mode hardware timestamping to clients. NTP’s multi-server redundancy and falseticker detection are preserved downstream, while the high-accuracy timestamping of the GPS layer is captured via PTP. The result: most servers can attest to being within 35–40 microseconds of UTC, well inside the 100-microsecond MiFID II requirement.

Content

Why clocks drift and why that matters

Every computer contains an oscillator — a small crystal — that vibrates at a nominally fixed frequency. The operating system counts those vibrations to maintain its sense of time. In practice the frequency is not perfectly stable: heat from a compiling job, an ageing crystal, or manufacturing variation can shift it by 50–100 parts per million, which translates to roughly a couple of seconds of drift per day on a typical server.

The consequence for a large network is that without active correction, two machines asked simultaneously what time it is will give different answers. For most applications — log correlation, Kerberos authentication (which requires clocks within five minutes) — tens of milliseconds of agreement is enough. For financial markets, it is not. European regulation MiFID II requires that certain participants, including Jane Street, demonstrate that every relevant clock is synchronised to UTC (Coordinated Universal Time — Greenwich Mean Time without time-zone or daylight-savings adjustment) within 100 microseconds. That is three orders of magnitude tighter than the default NTP accuracy most systems rely on.

How NTP works

NTP — Network Time Protocol, developed by David Mills — solves the offset-estimation problem with four timestamps. The client notes the time it sends a request (T1); the server notes when it receives the request (T2) and when it sends the reply (T3); the client notes when the reply arrives (T4). The round-trip delay is (T4 − T1) − (T3 − T2), and the clock offset is derived from the same four numbers — but only under the assumption that the packet took the same time in each direction. If the network is asymmetric, that assumption introduces a systematic error that no amount of averaging eliminates.

In practice NTP daemons query multiple servers, filter out outliers — termed falsetickers, with the honest servers called truechimers, nomenclature inherited from the original research — and use the remaining samples to estimate how much the local clock’s frequency should be adjusted. Crucially, well-implemented daemons do not snap the clock to the correct value, which would cause time to jump backwards and break any application that assumes clocks are monotone. Instead they slew: they instruct the operating system to run slightly fast or slow until the error is absorbed. Chrony, a modern NTP implementation, defaults to correcting one second of error over twelve seconds — a 10% rate change — which is fast enough for most recovery scenarios while remaining smooth.

PTP and its advantages

PTP — Precision Time Protocol — was designed to achieve sub-microsecond synchronisation and has three main advantages over NTP:

Hardware timestamping. When a PTP packet arrives at a network interface card (NIC — the hardware component that connects a machine to the network), the NIC can record the arrival time in hardware, before any operating system scheduling or interrupt latency adds jitter (random variation in timing). The timestamp reaches the application piggy-backed on the packet; the application never has to guess when the packet truly arrived. Many NIC vendors support this for PTP natively.

Higher update rate. PTP sends synchronisation messages every second by default; the NTP reference implementation’s minimum polling interval is eight seconds. More frequent updates mean errors are discovered and corrected sooner.

Switch participation. PTP was designed so that network switches — the devices that route packets between machines — can measure and subtract out their own queuing delays. This extends accurate timestamping across the entire network fabric, not just between endpoint and server.

On paper these advantages are compelling. In practice Chris Perl found that the switch-participation story created operational complexity: every firmware upgrade becomes a concern, and responsibility for timekeeping is spread across teams rather than concentrated in one place. The open-source NTP daemon chrony can also be tuned to hardware-timestamp NTP packets on cards that support it, and to poll as frequently as sixteen times per second, narrowing the gap considerably.

The robustness problem with PTP

NTP’s multi-server design is its strongest engineering property for a network like Jane Street’s. Each NTP client polls several servers, applies outlier rejection, and can route around a misbehaving source. PTP, by contrast, converges on a single grandmaster — chosen by the best master clock algorithm, which selects on clock quality rather than correctness. A GPS appliance that reports wrong time but claims high quality will be believed, and there is no protocol-level mechanism to call it out.

This was not theoretical. Jane Street had GPS appliances send wild timestamps when being shut down for maintenance, momentarily propagating nonsense to the network. A PTP-only architecture would have no defence.

Jane Street’s hybrid architecture

The solution Perl designed combines both protocols across two tiers.

Tier 1: GPS appliances → Linux servers via PTP. GPS appliances receive time from the satellite constellation at around 100-nanosecond accuracy (a nanosecond is a billionth of a second; by comparison, a microsecond is a millionth). They natively hardware-timestamp their PTP output but not their NTP output, so PTP is the right protocol for this leg. Each Linux server is paired with a single GPS appliance; if that appliance goes wrong, the server becomes a falseticker from the perspective of clients downstream.

Tier 2: Linux servers → clients via NTP with interleaved mode. Each Linux server acts as an NTP server with hardware timestamping enabled. NTP interleaved mode works around a limitation of hardware transmit timestamping: the NIC records the timestamp for a transmitted packet only after the packet has gone, too late to include it in that same packet. The interleaved extension sends the hardware transmit timestamp for the previous packet in the current packet, so the recipient always has an accurate hardware-stamped departure time one exchange behind. Clients see multiple NTP servers and apply standard falseticker detection; if one Linux server (and thus one GPS appliance) goes wrong, the others vote it out.

Clients are constrained to prefer their local NTP servers — ones close enough on the network that round-trip latency does not dominate — while also monitoring a set of distant servers as falseticker-detection references. A distant server cannot provide accurate time because its round-trip delay is too large and variable, but it can confirm that neither local server has gone completely off the rails.

Measuring and reporting compliance

The architecture is only half the work; the other half is demonstrating compliance. Each client tracks several error components: the known offset from its NTP server, the uncertainty accumulated while the daemon was not accepting updates (because it suspected a popcorn spike, for instance), and the round-trip delay divided by two (the minimum possible error from the symmetry assumption). Adding these together gives a conservative bound on how far the client might be from UTC at any moment.

On newer servers with client-side hardware timestamping, most machines achieve a worst-case bound of around 20 microseconds; on older servers without hardware timestamping, the bound rises to 35–40 microseconds. Both are well within 100 microseconds, but the margin shrinks noticeably when a NIC is saturated with other traffic. The solution is pragmatic rather than architectural: compliance is sampled every ten seconds and pushed to a central database, and alerts fire if any machine drifts out of bounds for a sustained period. Transient spikes from heavy NIC traffic are distinguished from persistent misconfiguration by the fact that they resolve quickly once the load subsides.

What comes after 100 microseconds

If regulations were to demand 10-microsecond accuracy — an order of magnitude tighter — the current architecture would not be enough. Perl points to White Rabbit, a set of PTP extensions that combine PTP with synchronous Ethernet (a standard that lets the physical Ethernet layer carry a clock signal, ensuring all devices on the network advance at exactly the same rate — syntonisation, as opposed to merely agreeing on the current time). White Rabbit, now partly incorporated into the latest PTP standard, achieves sub-nanosecond accuracy in laboratory settings. The reliability and redundancy engineering to make it safe for production would be substantial, and Perl notes that the decision not to pursue it was a function of the problem at hand: NTP, carefully tuned, was sufficient.

Related

• Chris Perl — speaker
• Ron Minsky — host
• Stephen Dolan on Memory Management, Garbage Collection, and OCaml — adjacent Signals and Threads episode on low-level systems engineering at Jane Street
• The Future of Software Engineering — theme; the conversation situates careful systems infrastructure work within the broader question of engineering at scale