Reinforcement Learning from Human Feedback

Reinforcement Learning from Human Feedback (RLHF) is a technique for improving Large Language Models in domains where correct answers cannot be verified automatically — creative writing, open-ended reasoning, tone. Human preference rankings substitute for an automatic reward signal.

Background: RL in verifiable domains

In maths and code, the model can generate many candidate solutions and check them against a known correct answer. It trains on the solutions that worked. This is clean RL: an automatic, reliable reward function. The model discovers reasoning strategies — including chains of thought — that no human explicitly wrote.

Limitations: this only works where answers can be verified. ‘Is this joke funny?’ has no automatic verifier.

The RLHF approach

Generate a set of candidate responses to a prompt.
Present 5 candidates to a human rater, who ranks them (ranking is easier than scoring or generating).
Train a reward model — a separate neural network — to imitate those human rankings.
Run RL against the reward model, querying it thousands of times per update step.

The reward model is a surrogate for human preference. It allows RL to scale without requiring a human in the loop for every rollout.

Discriminator-generator gap

It is much easier for humans to compare five generated poems than to write the ideal poem from scratch. RLHF exploits this asymmetry: humans label preferences, not examples.

The fundamental problem: reward hacking

RL is extremely effective at optimising for a given objective. The reward model is a neural network — a complex, imperfect approximation of human preference. RL will eventually find inputs that score highly on the reward model but are not genuinely good (adversarial examples of the reward model).

Run RLHF too long and the ‘best joke about pelicans’ becomes nonsense strings like ‘the the the the’ — which happen to score 1.0 on the reward model’s weird internal geometry.

Consequence: RLHF must be stopped early. It is used for a few hundred steps as a fine-tune — a polish — not as open-ended self-improvement. In verifiable domains, RL can run indefinitely; in RLHF, it cannot.

Karpathy: ‘sucking supervision through a straw’

Andrej Karpathy on AGI Timelines, RL's Limits, and the Future of Education sharpens the critique. Even in verifiable domains, outcome-based RL is wasteful: the model runs hundreds of rollouts, checks the final answer, then up-weights every token of the winning trajectories — including the wrong turns taken before stumbling onto the answer. A whole multi-minute trajectory is graded on a single final-reward bit, ‘broadcast across the entire trajectory.’ Karpathy calls this ‘sucking supervision through a straw’: ‘RL is terrible — it just so happens that everything we had before it is much worse.’ A human would neither run hundreds of rollouts nor credit every step equally; they would review which parts went well.

His account of why process supervision has not replaced this is the gameable-judge problem in concrete form: train RL against an LLM judge and the model finds adversarial completions — a string like ‘dhdhdhdh’ that the judge scores 100%. You can patch each one, but they are infinite, so the judge survives only ~10–20 RL steps. He expects ‘three or four or five’ more major algorithmic ideas before this is solved. See also his stronger claim that humans do not use RL for cognitive tasks at all — only for motor learning.

DeepSeek-R1 and the verifiable-domain breakthrough

The DeepSeek-R1 paper (early 2025) publicly described RL fine-tuning at scale for verifiable domains (maths, code). Key findings:

Accuracy climbs steadily over thousands of RL steps.
Average response length grows — the model learns that more thinking improves accuracy.
Chains of thought emerge without being programmed: ‘wait, let me re-evaluate this from a different angle’ is discovered by optimisation, not written by a labeller.

This is analogous to AlphaGo: RL discovered Move 37 — a Go move humans estimated at 1-in-10,000 probability that was in retrospect brilliant. RL in verifiable domains can discover strategies outside the human distribution.

Edwin Chen: post-training evolution and RL environments

Edwin Chen on AI Data Quality provides the most complete operational account of post-training methodology in the wiki. Edwin’s four-stage ladder:

Stage	Method	Notes
SFT	Supervised fine-tuning; imitate high-quality examples	Foundation; limits you to best human example
RLHF	Human preference ranking; reward model; RL	Scalable but subject to reward hacking
Rubrics and verifiers	Structured graded feedback with criteria	More signal per annotation; expensive to design
RL environments	Full real-world simulations; multi-step tasks; trajectory evaluation	Emerging; exposes gaps invisible in isolated benchmarks

RL environments are the emerging frontier: full simulations of real-world messy contexts (startup with Slack, Jira, GitHub, a codebase; then AWS goes down). Models must handle long-horizon tasks where step 1 affects step 50. Current models that perform well on isolated benchmarks fail catastrophically in these environments.

Trajectory evaluation is a key RL environments insight: checking only whether the model reached the correct final answer misses whether it got there via reward-hacking (random guessing until correct), inefficiency, or genuine reasoning. The path matters as much as the destination.

Each stage is additive, not sequential — fully trained frontier models use all four methods.

Karina Nguyen: synthetic data for product behaviour training

Karina Nguyen on Model Training and AI Product provides the most concrete account of how post-training is applied to specific product features.

The process for Canvas: identify three core behaviours (when to trigger, how to edit, how to comment) → write ground truth labels in a spreadsheet → use a stronger model (o1) to generate training examples → train the target model → measure with deterministic evals.

Why this extends Edwin Chen’s ladder: synthetic data is a technique that spans multiple rungs. For simple product behaviours, the synthetic data pipeline can replace human annotation entirely — o1 generates both the task and the ideal completion. For harder tasks (aesthetic quality, creative writing), human expertise remains necessary.

No data wall in post-training: Karina’s argument — the ‘data wall’ applies to pre-training on internet text. Post-training via RL has no comparable wall because any task is trainable. Benchmarks like GPQA (PhD-level Q&A) are saturating, but the number of tasks that can be added to post-training is infinite.

Deterministic product evals: for decision-making behaviours in product features, evals are binary and exact. If a user says ‘7:00 PM’, the model must output ‘7:00 PM’. No Likert scale; no subjectivity. These deterministic evals drive the synthetic training pipeline directly.

Brendan Foody: taste as an RLHF fallback, and RL environments as a labour-market mechanism

Brendan Foody on Teaching AI and the Future of Knowledge Work gives RLHF a second life beyond the pelican-joke reward-hacking failure mode above: as the deliberate fallback for domains where a rubric cannot be written at all. Pressed by Tyler Cowen with Kant’s argument (in the Critique of Judgment) that taste is precisely what a rule cannot capture, Foody concedes that a rubric works for a well-scoped essay but not for something as ‘unbounded’ as poetry, and names RLHF’s ranking mechanism — generate two candidates, have someone with strong taste choose the preferred one, repeat — as a genuinely different data type from a rubric, not a patch on one. This is the same discriminator-generator asymmetry the page already names above, applied for the first time in the wiki against an explicit philosophical objection to rubric-based training, rather than a technical one (see Evals for the parallel Altman/Cowen exchange on the same question).

Foody separately extends the RL environments row of Edwin Chen’s post-training ladder (above) with an economic mechanism for why they will proliferate. His claim: a large share of knowledge work is about to convert from recurring effort into a one-time teaching investment — an expert builds an RL environment once, encoding a task and how to judge success, after which an agent repeats it indefinitely at near-zero marginal cost, exactly mirroring software’s shift from bespoke repeated labour to a fixed cost copied at scale. The prediction this drives: a new job category — people whose work is to find where a model fails and convert that failure into a trainable RL environment — replacing, rather than merely displacing, today’s analyst-style knowledge work; Foody expects it to employ a majority of high-end knowledge workers within five years [?].

Practical implications

Thinking models (o1, o3, DeepSeek-R1, Claude’s extended thinking) are trained with RL on verifiable domains. Use them for hard maths, code, and reasoning; they are slower but more accurate.
Non-thinking models (GPT-4o, Claude 3.5 Sonnet) have had RLHF as a polish. Use them for factual queries, writing, and conversation; they are faster.