DeepSeek-R1

Checking an answer is easier than writing one

Take the equation 3x + 5 = 20. To check the answer x = 5, substitute it: 3 × 5 + 5 = 20. The check needs no worked solution. For a coding task, a set of tests can check whether a program returns the right outputs. Neither check says whether the explanation that came with the answer is sound.

A check makes a simple training loop possible: let the model attempt each problem several times and reward the attempts that pass. DeepSeek’s first experiment, R1-Zero, did exactly this. It ran reinforcement learning directly on the pretrained DeepSeek-V3-Base model, without first fine-tuning it on written reasoning examples. The rewards were simple: one for a correct final answer, and one for putting the reasoning between the required tags. The model still had its pretraining, the problems and the reward rules; only the usual stage of imitating example solutions was skipped. Over training, R1-Zero’s answers grew longer, and the paper reports that it began to re-check and revise its own work without being shown examples of doing so. [2]

The checker becomes part of what the model learns. A weak test suite rewards a program that fails on untested inputs, and a parser can misread an answer. The model optimizes the checker as written, not the intention behind it.

Comparing attempts at the same problem

The reinforcement-learning algorithm is Group Relative Policy Optimization (GRPO), introduced earlier in DeepSeek’s DeepSeekMath paper. [3] A raw reward is hard to interpret on its own: a correct answer to a trivial problem says little. GRPO judges each answer by comparison. It samples a group of answers to the same problem and scores each against the group’s average. PPO, the method used for Llama 2 and Tülu 3, trains a second network, a value model, to estimate that baseline. GRPO uses the group instead and drops the extra network.

Suppose four attempts receive rewards of 0, 0, 1 and 1. The mean is 0.5 and the standard deviation is 0.5. Subtracting the mean and dividing by the standard deviation gives each attempt’s advantage: −1, −1, +1 and +1. Training pushes the successful attempts up and the failures down.

Now choose “All equal” in the figure. Every advantage is zero, because the group offers no contrast to learn from. If all four attempts succeed, or all four fail, that problem teaches nothing at this step. Real implementations guard against dividing by zero here, and other parts of the training objective still apply.

What the update changes

A language model is a probability distribution over possible continuations. A positive advantage makes the tokens of that attempt more likely next time; a negative advantage makes them less likely. The full GRPO objective also limits how far the model can change in one step and penalizes drifting far from a reference model. The figure leaves out both. [3]

Say a careful solution path currently has probability 0.02. Each time an attempt along that path succeeds, training raises the probability of similar paths. Nothing installs a rule such as “always check by substitution”. If the model learns to check its work, that behavior emerges because attempts that checked tended to score higher.

Longer reasoning gives the model more room to try an idea, notice a contradiction and change course. It also costs more tokens and can include wrong steps. Length alone proves nothing, and the written reasoning is not guaranteed to reflect every step of the model’s internal computation.

How the final R1 was trained

R1-Zero and R1 are separate models. R1-Zero reasoned well but was hard to read: the paper reports poor readability and answers that switched between languages. R1 fixes this with a longer recipe. It starts from a few thousand readable examples of long reasoning, called the cold start, and then runs reasoning-focused reinforcement learning. [2]

Next comes a transfer through data. The reinforcement-learning checkpoint generates many answers, the authors keep the good ones, and they add general examples such as writing and factual questions: about 600,000 reasoning examples and 200,000 general ones. They then fine-tune the original DeepSeek-V3-Base for two epochs on this collection, starting again from the base model instead of continuing the earlier checkpoint. A final round of reinforcement learning covers both reasoning and general helpfulness. [2]

A student learning mathematics goes through similar stages: copy a few clear solutions, practice against an answer key, then study a curated collection of good attempts. The comparison describes only the kinds of feedback.

Smaller models trained on R1’s solutions

DeepSeek also released smaller models built on existing Qwen and Llama checkpoints. Each was fine-tuned on the 800,000 curated examples. These are different, smaller architectures trained on R1’s written output, not compressed copies of R1’s own weights. The repository documents the training data and notes that each small model also carries its base model’s license terms. [4]

This kind of distillation passes on what the teacher wrote: the student learns to produce similar solutions. The student does not see the teacher’s internal states or its probabilities for every token, as in the distillation used for Gemma 2. Both techniques are called distillation, but they transfer different information.

The release let researchers experiment with the large model and its compact descendants side by side. To compare them fairly, record each student’s base model, its training data, the sampling settings and how many tokens it was allowed to spend reasoning. Sharing the R1 name does not make those things equal.

Simplifications and limits

• R1-Zero starts from a pretrained model; the final R1 also uses supervised fine-tuning stages.
• The GRPO figure shows only the relative advantage, not the full training objective.
• The distilled models carry obligations from their Qwen or Llama base licenses.

Follow the idea

DeepSeek-V3DeepSeek-V3.2 / Exp

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Official DeepSeek-R1 releaseJanuary 20 release and downloadable original and distilled models.
2. DeepSeek-R1 technical reportSections 2.2–2.4: R1-Zero rewards, cold start and staged final recipe.
3. DeepSeekMath technical reportSection 4.1: GRPO origin, group baseline, clipping and KL terms.
4. Official DeepSeek-R1 repositoryDistillation dataset, downloadable checkpoints and license inheritance.