Sylvain Gugger on The Uncertain Art of Accelerating ML Models

Sylvain Gugger — machine-learning engineer at Jane Street, co-author of Deep Learning for Coders with fastai and PyTorch, and creator of Hugging Face Accelerate — joins host Ron Minsky to trace the full stack of performance engineering for ML training: from learning-rate schedules and GPU threading models to kernel fusion, mixed precision, data loading, and the peculiar challenges of trading data.

Key ideas

1. Training speed turned out to hinge on the learning-rate schedule, not just raw compute. The fastest entry in the 2017 DAWNBench competition used a warm-up schedule — starting low, climbing to a peak, then decaying — rather than the standard practice of training at a low rate and stepping it down. The schedule lets a randomly initialised model settle out of the wildest regions of a high-dimensional loss landscape before taking large steps, cutting training time to target accuracy without any extra hardware.
2. GPUs are massively parallel but fragile: any hidden synchronisation with the CPU destroys throughput. A GPU (Graphics Processing Unit — originally designed for rendering pixels, now repurposed for the repeated matrix arithmetic of neural network training) executes thousands of tiny programs called kernels in parallel, fed by the CPU asynchronously. The moment Python code needs a value back from the GPU — to check whether a loss is NaN, for instance — the CPU stalls, the GPU queue drains, and throughput collapses. Profiling to find and eliminate these stalls is the first step in any performance investigation.
3. Kernel fusion removes two costs at once: launch overhead and intermediate memory traffic. Each operation in a model dispatches a separate kernel, which takes microseconds to start and writes its result to GPU memory for the next kernel to read. Fusing consecutive kernels into one bigger kernel eliminates both the launch overhead and the round-trip through memory, because intermediate values can stay in the GPU’s fast on-chip registers. PyTorch’s torch.compile automates this by capturing the full computation graph and generating fused kernels via Triton — a Python-like domain-specific language that compiles down to efficient CUDA (NVIDIA’s low-level GPU programming language).
4. Mixed precision cuts compute time two- to four-fold by exploiting specialised matrix-multiply hardware. Modern GPUs have dedicated Tensor Cores built specifically for 16-bit floating-point matrix multiplies — the dominant operation in transformer training. Running weights in 16-bit (float16 or bfloat16, a Google variant designed for stability) rather than the standard 32-bit format is usually sufficient for layer activations as long as the final accumulation stays in 32-bit, and the speedup is near-free. NVIDIA’s newest chips push this further to 8-bit and even 4-bit formats.
5. Trading data breaks the assumptions baked into standard ML tooling — noisier signal, vastly larger volume, and a non-stationary target. Language models improve on data from 2018 as reliably as data from today; a model trained on market data from two years ago may perform no better than random, because profitable strategies get arbitraged away once they become known. At Jane Street, this means continuously reinventing model architectures, custom data-loading infrastructure sized for terabytes per day, and inference latency targets measured in microseconds rather than seconds — none of which the major ML frameworks were designed for.

Content

From maths teacher to DAWNBench

Gugger came to machine learning circuitously — a decade teaching first-year university mathematics in France, then a move to the United States, then a 2017 New York Times article about AI that led him to Jeremy Howard’s fast.ai course. His increasing contributions to the fast.ai library brought him onto the team, and the first real test was the DAWNBench competition: train a computer vision model to a given accuracy as fast as possible.

The fast.ai team’s two winning techniques were both departures from convention. First, the warm-up learning-rate schedule described above. Second, progressive resizing: convolutional neural networks (CNNs — models built for image data, applying the same filter at every position rather than learning separate weights for every pixel) are resolution-independent, so training begins on small 128×128 images and increases resolution as training matures. A randomly initialised model does not need full-resolution images to learn basic structure, and the smaller inputs process faster. At the end, Google dropped TPUs (Tensor Processing Units — Google’s custom AI chips) into the competition and took first place, but the fast.ai approaches had already demonstrated that schedule and data choices mattered as much as hardware.

Hugging Face and the problem of configurable training

After fast.ai, Gugger joined Hugging Face — which he describes as ‘the GitHub of machine learning’: a platform for sharing model weights alongside the code needed to instantiate them. His main technical contribution there was the Accelerate library, born from the frustration that the existing Trainer API had accumulated so many flags and edge-cases that it had become a wall of spaghetti code.

Accelerate’s design principle is minimal intrusion: a researcher writes their training loop normally, then changes a handful of lines to make it run on one GPU, eight GPUs, or a TPU cluster, without any other modification. Where a compiled framework like TensorFlow or JAX hides hardware decisions from the user, Accelerate keeps the researcher in control of where data lives and when it moves — a trade-off that sacrifices some automatic optimisation in exchange for the ability to reason about and fix the code. The parallel to OCaml’s memory-management philosophy (opt-in control rather than mandatory annotation) is one Ron Minsky draws explicitly.

The parallelism strategies Accelerate had to support ranged from data parallelism — giving each of n GPUs a different slice of a batch, reducing training time by roughly n — to tensor parallelism, where even the weight matrices of a single model are split across cards, requiring constant high-speed communication. NVIDIA’s NVLink fabric and eventually cabinet-scale InfiniBand interconnects exist specifically to serve that communication need; the physical placement of GPUs has become a variable in model performance.

PyTorch’s unlikely dominance

Both Ron Minsky and Gugger find PyTorch’s success somewhat against first principles. TensorFlow and JAX express the model as a compiled computation graph, which makes it easy to guarantee performance — you cannot accidentally write something the compiler cannot optimise. PyTorch executes eagerly, dispatching individual Python lines to the GPU one at a time, which means performance depends on the programmer’s discipline. Yet PyTorch won, for two reasons.

First, researcher ergonomics: if an idea is wrong, the fast feedback loop of eager execution is worth more than the efficiency of a compiled run. Second, dynamic shapes: training on variable-length sequences — where each batch may be a different shape — is natural in PyTorch but requires awkward workarounds in JAX or TensorFlow, which expect all inputs to have fixed shapes. The industry eventually reached PyTorch’s solution anyway, with torch.compile added in version 2.0 to bring graph capture and kernel fusion to the eager world.

Profiling and the kernel fusion workflow

Gugger’s day-to-day role at Jane Street is reactive: a researcher arrives with a training run that is slower than expected, and the first step is always profiling. A profiler shows the timeline of GPU and CPU activity — in particular, the synchronisation stalls where the CPU waited for the GPU to return a value before it could schedule the next operation. Gugger gives the example of a NaN check: to branch on whether the loss is NaN (not-a-number — a signal that training has gone catastrophically wrong), the CPU must inspect the GPU’s result, which forces a stall. One solution is to move the check to a background thread so the main training loop is never blocked.

Once synchronisation stalls are eliminated, the next step is torch.compile, which hands the graph to Triton for kernel fusion. The resulting fused kernels keep intermediate values in fast on-chip registers rather than writing them to and reading them back from the GPU’s main memory (VRAM — the GPU’s equivalent of RAM, slower to access than registers). CUDA graphs offer a lighter alternative: they replay a pre-recorded sequence of kernel launches without re-engaging the CPU, eliminating launch overhead without the memory savings of true fusion.

CUDA streams add a further dimension: they allow two independent sequences of GPU work — for instance, transferring the next batch of data onto the GPU while computing predictions on the current batch — to run in parallel, so the GPU is never idle while waiting for data.

Mixed precision and the precision ladder

Most of the speedup from moving from 32-bit to 16-bit arithmetic is effectively free for standard deep networks. The weights of both matrices in a matrix multiply can be stored in float16 or Google’s bfloat16 variant (the ‘b’ stands for ‘brain’) while the accumulated result stays in 32-bit, producing correct results at half the memory bandwidth and unlocking the GPU’s Tensor Cores. Newer NVIDIA hardware — the Blackwell generation — supports 8-bit and possibly 4-bit formats, each step further reducing bandwidth cost as the GPU hardware specialises to compensate for the reduced precision.

The ML language ecosystem

CUDA is the foundational layer: a C-like, proprietary NVIDIA language that exposes the GPU’s threading model directly. Threads on a GPU are unlike CPU threads — rather than a handful of cores switching context, a GPU may run a million threads simultaneously, organised into warps of 32 that execute the same instruction in lockstep. A streaming multiprocessor (SM) is the physical block that schedules warps, shares fast on-chip memory among them, and hides memory latency by swapping in a ready warp the moment another stalls on a memory fetch. CUDA exposes this model explicitly but unsafely, with pointer arithmetic, undefined behaviour, and sparse documentation.

Above CUDA, Triton offers a Python-like syntax that compiles to efficient CUDA kernels. It is well-suited to operations that look like matrix multiplies and reductions — the bulk of neural network arithmetic — but fragile on anything outside that envelope. Mojo (still maturing at the time of recording) occupies a similar tier but as its own language rather than a Python DSL, with stronger type-safety guarantees. Numba takes the other trade-off: CUDA semantics in Python syntax, for programmers who want low-level control without leaving the Python ecosystem.

Trading data as a separate domain

The closing section of the conversation turns to what makes Jane Street’s ML environment different from the canonical cases the tools were designed for. Trading data is noisy by construction: prices approximate a random walk because any exploitable pattern gets traded out of existence. The signal-to-noise ratio is so low that a model capturing even a small fraction of predictable variation can be valuable, but that fraction is never stable — strategies must be continuously refreshed.

Volume is extreme: several terabytes of market data per day, compared to the static corpora most NLP or vision models train on. This stresses data-loading infrastructure that was designed for well-shuffled, same-size batches of images or text tokens; at Jane Street, data-loading is a first-class engineering problem with substantial custom code.

Inference latency requirements span orders of magnitude: from ‘once an hour’ for some analytical tasks to under a microsecond for the fastest trading decisions. The smallest, most latency-sensitive models sit entirely outside the design envelope of NVIDIA’s optimised kernels, which are tuned for the large batches and large model sizes typical of language-model training.

Reproducibility, Gugger adds, is particularly acute in this environment. Machine learning already struggles with reproducibility — GPU non-determinism from parallel floating-point accumulation, untracked package versions, notebook cells run out of order — and market data adds the complication that a model reproducing last year’s results may simply be replicating a strategy that no longer works.

Related

• Sylvain Gugger — speaker
• Ron Minsky — host
• Stephen Dolan on Memory Management, Garbage Collection, and OCaml — adjacent Signals and Threads episode; Ron Minsky draws a parallel between OCaml’s opt-in memory control and PyTorch’s explicit GPU/CPU data placement
• Chris Lattner on Why Machine Learning Needs a New Programming Language — overlapping territory on the ML programming-language ecosystem, including Triton and Mojo
• The Future of Software Engineering — theme; the conversation situates ML performance engineering within the broader question of what control programmers should have over execution
• Large Language Models — concept; the episode frequently grounds GPU optimisation discussion in transformer and LLM training