Jacob Baskin on Building a Data Warehouse from Scratch

Jacob Baskin — a Jane Street software engineer whose career runs from Google’s advertising exchange through a curb-management startup — joins host Ron Minsky to trace how he came to build Superstore, Jane Street’s from-scratch analytical data warehouse, and now works on The Hive, its compute cluster. The through-line is mechanism design: the economics of getting people to say what they want, and the discovery that explainability and simplicity matter far more than theoretical elegance.

Key ideas

1. Transactional and analytical databases pull in opposite directions. A transactional database (Postgres is the canonical example) is built to change one row at a time, atomically and fast — the checkout-button problem, where you record an order and decrement inventory together in the moment the page loads. That requirement forces a data layout — row-by-row storage, on-disk indexes, no compression across rows — that is exactly wrong for analytical work, where you want to sweep millions of rows to answer one aggregate question. Postgres will do analytics, but slowly, and only on data that fits on a single machine. Superstore exists because Jane Street outgrew that single machine.

2. Column storage buys speed but breaks the transactional contract. A columnar database stores each column as its own compressed stream, so a query touching seven of a hundred columns ignores the other ninety-three and adding a column is nearly free. The cost: updating a single row now means touching many separate, compressed streams. Off-the-shelf columnar systems (Jane Street ran Vertica) try to hide this behind full SQL transactional semantics, but the abstraction leaks — ‘use it just right and it’s efficient’, which is precisely what a Wild-West user base will not do.

3. Superstore’s central bet: give up transactions between tables, keep them within a table. The trades-of-the-day log needs never to miss or double-count a trade, but it does not need to be written atomically with unrelated data. So Superstore offers atomic, order-preserving writes to a single table and no consistency guarantees across tables — enough for a workload that is mostly appends from a known source of truth. A second unusual choice follows: all writes are asynchronous. The database acknowledges (commits) a write before readers can see it, using the gap to place the data efficiently in columnar form and redo compression — paying the cost neither while a writer waits nor while a reader waits, but in between.

4. The requirement was to preserve the ‘Wild West’. The prize of Jane Street’s shared databases is that traders and researchers can throw almost any data in and run almost any query — ‘database as ecosystem’, where each new table joins with all the others and raises the platform’s value. Many firms tame the same problem by killing that flexibility: schema councils that sit in judgement before you add a table, databases where only code-reviewed queries may run. Superstore’s brief was the opposite — keep the freedom, but make it much harder for one user to break the system for everyone, in both correctness and performance.

5. Being in-house is a design advantage, not a constraint. Because every user works at Jane Street, the team can educate them — teaching the asynchronous-write model directly and writing the first examples themselves — rather than shipping semantics that need no explanation. It can also refuse to let users read the underlying Parquet files directly (a request ‘always denied’), because routing every read and write through the system is what makes access control, usage logging, and — crucially — the ability to deprecate a table or evolve a schema possible at all. The much-derided not-invented-here instinct is defensible here: the open-source community understands the website-clickstream use case, not Jane Street’s, and a competitive edge lives in doing the finance-specific thing well.

6. The regret is the metadata store. Superstore’s metadata — what tables exist and where their data lives — must be globally consistent, so the team chose CockroachDB, a distributed transactional database. It scales by splitting rows across many independent consensus groups, but repeated writes to the same row serialise through a distributed consensus round each time, and that single-table commit rate — not query throughput, not total commit throughput — is the bottleneck streaming data hits. The fix underway is Aria, a finance-style state-machine-replication system that gets consistency by funnelling everything through a single sequencer. That reintroduces a single point of failure, but Baskin’s defence is that a single single point of failure, on a well-tended machine, means roughly a minute of read-only downtime every few years — and that real outages are almost always software bugs, not the hardware failures elaborate designs guard against.

7. The Hive turns scheduling into mechanism design. Baskin now works on The Hive, Jane Street’s compute cluster — hundreds of thousands of CPU cores and over ten thousand GPUs, big enough to accidentally denial-of-service almost anything it points at. Jobs are allocated by a second-price auction in dollars per core-hour, but the mechanism cannot capture the dimension that matters most: urgency. A $5,000 job that is worthless in an hour should beat a $10,000 job that is just as valuable tomorrow. The right object is not a single price but a utility curve over time — and what matters is its derivative — which for a periodic monitoring job is not even convex (there is a peak around the ideal run time). Modelling this makes an already NP-hard gang-scheduling problem (whole groups of tightly-synchronised GPUs, topology-aware, across data centres) harder still, but the better schedules are worth it.

8. Everything converges back to SQL and the query planner. A database’s query planner is a compiler that turns SQL into a program — SQL only hides this because it is such a good abstraction that asking a question does not feel like writing one. The same insight reappears at every layer: BitTorrent distributes a job’s binary to a thousand workers by making receivers into transmitters; data-frame libraries like Polars and engines like Spark win by lazy execution — building the whole query before running it so the system can optimise it. The Hive’s future is a declarative computation graph: the more the system knows about what you are trying to run over what data, the better it can schedule and distribute it.

Content

From mechanism design to a two-sided market

Baskin planned to study psychology, but could not stay away from coding, and wrote a Brown thesis on mechanism design — the branch of economics concerned with designing rules (an auction is the archetype) so that participants do best by telling the truth. He expected never to use it. His first job, at Google in 2008, put him on AdExchange, the product Google built after acquiring DoubleClick to show ads on other companies’ websites. It was a two-sided market — publishers with space, advertisers with ads — and it ran on real-time bidding: for every ad impression, Google queried advertisers’ systems across the internet, collected bids, and, for the winner, injected an HTML snippet that fetched the image directly from the advertiser. ‘Just in time twice.’ DoubleClick’s contribution was not technology but an understanding of enterprise customers — the DNA of serving large publishers and advertisers, which Google, with its ‘why would anyone want to talk to a salesperson’ ethos, lacked.

What incentive-compatibility misses

The academic message Baskin absorbed was that building incentive-compatible mechanisms — where honesty pays — is the valuable thing. Industry taught him it is a much smaller part of the problem than he had believed. Two things the theory omits dominate in practice. First, people have to understand the mechanism: a proof that it is incentive-compatible is worthless if the user cannot see why they were charged what they were charged. Second, simplicity is a value in itself. His example is budgets. An advertiser with a daily budget who is spending too fast should rationally bid below their true marginal value to make the money last — which breaks the clean, memoryless auction. Rather than model each participant’s budget for them (and hand back results they cannot follow), real systems let people manage their own budgets and stay incentive-compatible only in a narrower, legible sense. People carry constraints they never tell you; the humane design lets them.

Google’s distributed-systems doctrine

Baskin says he is ‘still mining’ his Google years for engineering lessons. The central one: contain the hard part. Take an existing system that solves one genuinely difficult distributed problem — consensus, or a large key-value store — and build on top of it, often in nested layers (BigTable on Chubby; most engineers building on both). Keep as few components stateful as possible, because large stateless systems are far easier to build than stateful ones: a stateful core loads data into memory in an efficient form, and a swarm of stateless web servers does the rest. This shape does not fit trading, where latency pushes state out to the edge — Google’s ad-serving latency budget was 250 milliseconds, enough that you would not go for coffee waiting, but worlds away from the microseconds trading systems chase. Organisations, Minsky notes, tend to be shaped like their first application: Google’s infrastructure grew around search (ads were treated as ‘just a search problem’), Jane Street’s around trading.

The curb-management interlude

Between Google and Jane Street, Baskin did a startup helping cities manage their curbs — decoding the accreted thicket of parking signs that determine, block by block and hour by hour, what a vehicle may do. Positioning signs to within one parking space is tighter than urban GPS allows, so they used augmented-reality-style visual odometry (the Pokémon GO trick of building a map of a scene inside the phone) to locate signs along a block, then hand-built parsers for the small but spatially strange language of parking signage. He loved the problem; it did not make a business. Wanting economic thinking, New York, and a place where the people he respected had already congregated — many of them writing OCaml — he interviewed at Jane Street.

Why Postgres was not enough

At his team-match, the database-infrastructure team described the pain of running very large Postgres databases into which people freely threw data. Baskin’s opening move was to announce he would build a distributed analytical warehouse instead. The diagnosis: Postgres is a very good database serving goals that fight the analytical use case. Its transactional guarantees force a layout — indexes on disk, no cross-row compression, whole-row updates — optimised for changing few rows fast, not for reading many rows at once. And it keeps all data on one machine; Jane Street’s largest computer was the one running Trader DB. Worse, the shared, undisciplined, high-value use pattern generated recurring performance crises. Long-running transactions were the worst offenders: Postgres gives transactional isolation by locking every row a transaction writes (a segmented, copy-on-write fork of the touched pages), so a long transaction accrues locks, blocks others, and impedes replication and vacuuming. Jane Street ran a long-running-transaction killer and a deadlock detector that simply killed the offenders.

Superstore’s bets

The requirements: keep the Wild West, store more data than fits on one machine, compress far better, read large spans efficiently — and make it much harder for one user to break isolation. Something had to be given up, and the candidate was transactions. The landing point sat between systems that abandon transactions entirely (Kafka, which lacks read-modify-write and scopes ordering to a single partition; Redis, with no guarantees across keys) and Vertica, which offers full semantics but only performs if used just right. Superstore keeps atomic, order-preserving writes within a single table and drops guarantees between tables — sufficient because most changes are appends pulled from a source of truth. Writes are asynchronous: acknowledged before they are readable, so the system can lay the data out in columnar form and compress it in the interval. SQL is the first-class read interface but need not be the first-class write interface — reads were ad hoc and human-written, but writes mostly came through software systems. Single-row writes are simply not offered; ad hoc data is uploaded a whole table at a time and re-uploaded to change. And the underlying Parquet files, though sitting in a distributed store, are deliberately not exposed for direct reading — because mediating every access is what buys access control, usage logging, and the ability to answer ‘who is still using this table, and can I delete it?‘

The metadata regret, and Aria

Baskin’s stated regret is metadata. The metadata store — the pointers to where each table’s data lives — must be transactional and globally consistent (you cannot have consistent data behind inconsistent pointers), so they chose CockroachDB. It was a good fit in many ways, but write latency at scale was not: the majority of Superstore’s bytes arrive streaming, wanting to materialise within a second, and each write is a transactional update that, for a distributed system, means a consensus round. CockroachDB scales by dividing rows across many independent Raft consensus groups — but hammering the same row serialises those rounds, and a single round’s wall-clock time is high. The replacement is Aria, which achieves consistency by shoving all data through a single sequencer and distributing it by reliable ordered multicast. That is a genuine single point of failure — one plug whose pull pauses a whole cluster — but recovery is a controlled flip to a replicated backup, leaving the cluster read-only rather than dead. Baskin’s argument for accepting it: CPUs are fast, and it is network communication and synchronisation that make consensus slow; a single point of failure on a well-cooled machine costs about a minute every four years; and downtime is dominated by software bugs anyway.

The Hive: scheduling as economics, and the road back to SQL

Baskin now works on The Hive, the cluster where Jane Street runs work needing many computers at once — neural-network training (on GPUs), feature-data production and large historical trading simulations (on CPUs). Growth is driven by machine learning. At this scale The Hive is ‘a giant DOS machine’ that can knock over any storage appliance it feeds from; a favourite footgun is thousands of machines creating files in one NFS directory at once, since creating a file is effectively writing to the directory. The team isolates The Hive from trading systems entirely and builds framework APIs that stop users from touching storage in dangerous ways.

Two harder problems shape the current mission. The first is scheduling as mechanism design: jobs bid in dollars per core-hour under a second-price auction, but the auction happens when the job is submitted and cannot express urgency, so people schedule for the small hours to catch cheap resources and build Python notebooks to model future demand — complexity pushed onto users. The fix is to let people hand over a utility curve over time and let the system pick the moment, though designing that API without turning job submission into an economics quiz is itself hard (and the curves are not always convex). The second is running work efficiently on each worker: getting a large binary onto a thousand cores without melting the one machine that holds it, solved with BitTorrent — receivers become transmitters, evening out the load. Coming to The Hive was the opposite of building Superstore from scratch: the system already existed, twenty years deep, actively used, so Baskin began by sitting on a trading desk for a month, using The Hive alongside researchers to learn how features shipped in isolation failed to reach users buried under other layers. The endpoint he is building toward is a declarative computation graph — a distributed query engine, Polars-and-Spark-shaped, privileging the finance operations (over dates, over symbols) Jane Street understands better than any vendor.

Related

• Jacob Baskin — speaker
• Ron Minsky — host
• The Future of Software Engineering — theme; Superstore’s argument that the load-bearing skill is choosing which trade-offs to make, and for whom, sits squarely inside it
• Andrew Hunter on Performance Engineering on Hard Mode — the sibling Jane-Street-engineering episode; latency shaping architecture, and the primacy of doing less work
• Doug Patti on State Machine Replication, and Why You Should Care — the consensus-versus-single-sequencer trade-off Baskin invokes when replacing CockroachDB with Aria
• Utility Curves — Stewart Butterfield’s product-strategy model; Baskin reaches independently for a utility curve to argue that a job’s urgency (the curve’s derivative), not a single price, is what a scheduler needs