Gemma 2 · 9B and 27B

How a larger teacher model gave a smaller student more to learn from each token

A training document shows a model only the word that actually came next. It says nothing about the other words that would have worked nearly as well. Gemma 2, released by Google in June 2024, trained its 9-billion-parameter model on a richer signal: at every position, a larger teacher model’s probabilities for every possible next token. This page explains that technique, called distillation, and the attention design that made Gemma 2 cheaper to run.

5 sections · about 8 minutes · Training

Release: 2024-06-27. Google released the 9B and 27B weights on June 27, 2024. The date excludes the earlier preview and the 2B model, which came later.Model: 9B distilled · 27B conventionally pretrainedWeight license: Gemma Terms of Use; custom terms and use restrictions

What one observed word leaves out

Google released Gemma 2 in 9-billion and 27-billion-parameter sizes on June 27, 2024; a 2-billion version followed later. Most of the architecture combined techniques already in use elsewhere. The distinctive choice was training the smaller model by distillation from a larger one. [1]

Take the fragment “She poured the tea into the”. Suppose the document continues with “cup”. Ordinary training rewards the model only for the probability it gave “cup”. Yet “mug” would also have been a good continuation, and far better than “volcano”. A target containing only “cup” treats “mug” and “volcano” as equally wrong.

A larger model that has already been trained can supply the missing information. Its prediction is a probability distribution over every possible next token: perhaps 60% “cup”, 30% “mug”, a little for “bowl” and almost nothing for “volcano”. Training a smaller student model to match such a teacher distribution is called knowledge distillation. The teacher can be wrong, and its errors and biases pass to the student along with what it knows.

Softening the teacher’s prediction

Distillation is older than Gemma; Hinton and colleagues described it in 2015. [2] A model first produces a raw score for each token, called a logit, and then converts the scores into probabilities. Dividing the scores by a temperature before converting them changes how peaked the result is. At temperature 1, the figure’s made-up scores give “cup” 83% and “mug” 11%. At temperature 2, “cup” falls to 58% and “mug” rises to 21%, so the alternatives become easier to see.

Higher is not always better. At very high temperatures every token approaches the same probability, and the distinctions vanish again. Hinton and colleagues raised the temperature to expose how the teacher ranked unlikely answers. The figure shows that general effect, not Gemma 2’s settings.

The student does not copy the teacher’s weights. It adjusts its own weights so its predictions move toward the teacher’s distribution. The student can have a different architecture, and once trained it runs on its own, without the teacher. [2]

See the alternatives in a teacher’s prediction

Made-up scores for four candidate words. The figure shows distillation from token probabilities; it does not reproduce Gemma 2’s training settings. A soft target shows how the alternatives compare, not only which one won.

Why matching the whole distribution helps

Suppose the teacher assigns 0.6 to “cup”, 0.3 to “mug” and 0.1 to “bowl”. One student predicts 0.6, 0.3 and 0.1, matching the teacher. Another predicts 0.6, 0.01 and 0.39. Both give “cup” the same probability, so ordinary training, which looks only at “cup”, scores them equally. A loss that compares whole distributions prefers the first student, because it ranks the alternatives the way the teacher does.

The Gemma 2 report trained the 9B model this way throughout pretraining. Its loss weights the student’s log-probability for every token by the teacher’s probability for that token. The 27B model was trained conventionally, on the observed next tokens. The report also tests distillation in smaller controlled experiments. [3]

The motivation is efficiency. Small models are now trained on far more tokens than their size alone would call for, and each further pass over ordinary text teaches less than the last. Distillation gives the student more information at each position instead of more positions. It guarantees nothing: the teacher’s quality, the student’s size and the length of training all limit what the student learns.

Alternating local and global attention

Gemma 2 also lowered the cost of running the model, using two techniques already established elsewhere. Its layers alternate between local attention, where each token sees only the previous 4,096 tokens, and global attention, where it sees the full 8,192-token context. A local layer needs to cache only those last 4,096 tokens. The model also uses grouped-query attention, in which several attention heads share stored keys and values. [3] These changes are separate from distillation: distillation affects how the model is trained, and these affect what it costs to use.

The figure shrinks the idea to 16 tokens. Choose a query position near the end and switch between Local and Full. The highlighted cells are the earlier positions that a single layer can attend to directly. A local layer sees a fixed-width neighborhood; a full layer sees everything before the query.

Because Gemma 2 alternates the two, every other layer can reach the whole context. Reaching a token is not the same as using it well, though. The figure shows which connections are allowed, not which ones the trained model relies on.

Compare a local layer with a global layer

Simplified attention pattern for one layer, with 16 tokens. Local and Full show Gemma 2’s two layer types; “Toy hybrid” is a generic teaching pattern, not Gemma 2’s layer order. Local layers limit what each layer sees directly; the global layers between them keep long-range access.

Two meanings of “distillation”

The model card lists separate pretrained and instruction-tuned checkpoints under Google’s Gemma license. Instruction tuning is a later training stage that shapes how the model responds to people, so results for the base model and the chat model should not be mixed in a comparison. [4]

“Distillation” names two different techniques. In the one described here, the student learns from the teacher’s probabilities at every token. In the other, the teacher writes complete answers and the student is trained to imitate them as ordinary text. The first passes on the teacher’s view of every alternative; the second passes on only the words the teacher chose. When a model card says a model was distilled, check which kind it means.

Remember the two students that both gave “cup” 60%. Once training compares whole distributions, the model is rewarded for more than the one word the document happened to contain.

Simplifications and limits

The release date covers the 9B and 27B models; the 2B model came later.
The temperature figure uses made-up scores and does not reproduce Gemma 2’s training settings.
The attention figure is a scaled-down illustration, not a trace of the model running.

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

Gemma 2 is now availableJune 27 release of 9B/27B, download availability and licensing context.
Distilling the Knowledge in a Neural NetworkSection 2: soft targets and temperature; the widget uses invented scores.
Gemma 2: Improving Open Language Models at a Practical SizeSections 2, 3.2 and 5: local/global layers, teacher-probability loss and distillation ablations.
Google’s Gemma 2 9B model cardCheckpoint purpose, released weights and Gemma terms.