Llama 2 · 7B, 13B, 70B

What a base model learns

Meta released Llama 2 on July 18, 2023, in three sizes: 7, 13 and 70 billion parameters. Each size came as two sets of weights. The base model was trained only to predict text. The chat model started from the base model and was trained further to answer people. Because Meta published both, anyone could give the same prompt to each and see what the extra training changed. The weights were free to download under Meta’s own community license, which permits commercial use but adds conditions that a standard open-source license does not. [1]

A base model learns from one exercise repeated across about 2 trillion tokens of text, where a token is a word or a piece of a word. [2] The model reads some text and assigns a probability to every token that could come next. After “The train leaves at”, it might give high probability to “7” and “noon” and low probability to “purple”. Training compares those probabilities with the token that actually followed and adjusts the weights to make that token more likely next time.

The result models how text continues, which is a different skill from helping someone. Give a base model “What is the capital of France?” and a likely continuation, learned from quiz sheets, is “What is the capital of Spain?” Turning the predictor into an assistant needs a different training signal: examples of good answers, and judgments about which answers are better.

How the chat model was trained

Meta trained Llama 2-Chat in two stages. [2] The first was supervised fine-tuning. People wrote prompts paired with good responses, and the model was trained to reproduce those responses. This teaches the shape of an answer: address the request, then stop.

Written examples cannot cover every prompt, and writing excellent answers is slow. Judging is faster. An annotator reads two responses to the same prompt and picks the better one. Meta collected these comparisons and used them to train two reward models, one for helpfulness and one for safety. A reward model is a separate network that reads a prompt and a response and outputs a score predicting how annotators would rate it.

The reward models then guided the chat model over several rounds. In rejection-sampling fine-tuning, the chat model writes several answers to each prompt, the reward model scores them, and the chat model is trained on the best one. In proximal policy optimization (PPO), a reinforcement-learning method, the chat model’s weights are adjusted directly to raise its reward, while each update is limited so the model does not change too far at once. Early rounds used rejection sampling alone; later rounds used both. None of these methods was new. What Llama 2 added was a detailed public account of how to combine them. [2]

Consider three answers to “What is 17 × 24?”: one correct with clear working, one correct but muddled, and one fluent but wrong. Predicting the next token of web text gives no reason to prefer the first. A comparison does, because an annotator ranks the three and the reward model learns that ranking.

The reward model only approximates the annotators, so it can be fooled. A wrong answer that sounds confident and well organized may still score highly. A higher reward means the answer looks more like what annotators preferred; it does not prove the answer is correct.

Why a long conversation uses memory

The second change affects the model while it writes. A chat model produces its reply one token at a time, and each new token looks back at everything before it. The mechanism that does this is attention. The new token produces a query, a list of numbers describing what it is looking for. Every earlier token has a key, which is compared with the query, and a value, the information passed along when the comparison matches well.

An earlier token’s key and value do not change when later tokens arrive, so the model computes them once and stores them. This store is the KV cache. It saves a great deal of recomputation, but it grows with every token, it exists in every layer of the model, and a server needs a separate one for each conversation it is handling.

Attention runs as several heads in parallel, each with its own query, so different heads can look for different things. In standard multi-head attention, each head also has its own keys and values, and the cache stores all of them. Grouped-query attention (GQA), proposed by Ainslie and colleagues in 2023, lets a group of query heads share one key and one value. With 32 query heads in groups of four, the cache holds 8 key–value pairs per token per layer instead of 32. The queries stay separate, so each head can still look for something different. Multi-query attention (MQA) is the extreme case, in which all heads share a single pair. [3]

Llama 2’s 70B model uses grouped-query attention, which the paper adopted to make its largest models cheaper to run. The 7B and 13B models use standard multi-head attention. The paper also describes a 34B model, but Meta did not release its weights. [2]

Working out the cache size

The figure multiplies six numbers. For 4,096 tokens with 8 key–value heads: 2 (one key and one value) × 32 layers × 4,096 tokens × 8 heads × 128 numbers per head × 2 bytes per number = 536,870,912 bytes. That is 0.50 GiB in the binary units the chart uses, or about 0.54 GB in decimal units. If each of the 32 query heads kept its own key and value, the cache would be four times larger: 2 GiB.

Because every factor multiplies, doubling any one of them doubles the cache. Move the token slider from 4,096 to 8,192 and both bars double. Choose MQA and the cache becomes eight times smaller than the grouped setting. The chart cannot show what that sharing costs in answer quality. The GQA paper measured it and reports that grouped attention stays close to the quality of full multi-head attention, while a single shared pair loses more. [3]

Llama 2 was trained with a 4,096-token context window: the most text, prompt and reply together, that it handles at once. [4] The two kinds of memory behave differently. The weights are fixed and shared by every conversation on a server. The cache belongs to one conversation and grows as that conversation continues.

Which saving matters depends on the job

A server answering many short chats spends most of its memory on the weights, which all the chats share, so a smaller cache helps only a little. A server reading one long document builds a large cache for that document, and sharing keys and values frees a lot of memory. Even then, a cache four times smaller does not make the model four times faster, because storing the cache is only one of several costs in producing a token.

Llama 2 is a good starting point because it separates three questions that later models answer in different ways: what a model learns from raw text, how further training turns it into an assistant, and how much memory it needs to keep a conversation going. Later pages change these one at a time. Keep them apart when you read a release announcement. A better chat model does not imply a new attention design, and a smaller cache does not imply better answers.

Simplifications and limits

• The memory figure uses a made-up model shape and counts only the KV cache. It does not predict speed or answer quality.
• The training description follows the paper’s overview. It omits details such as Ghost Attention, the paper’s method for keeping a system instruction in effect over many turns.
• Llama 2 used grouped-query attention and reinforcement learning from human feedback; it invented neither.

Follow the idea

Llama 3.1 · 8B, 70B, 405B

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Meta and Microsoft introduce Llama 2Release date, downloadable base/chat models and licensing context.
2. Llama 2: Open Foundation and Fine-Tuned Chat ModelsSections 2, 3.1–3.2 and Appendix A.2.1: model variants, GQA and chat pipeline.
3. GQA: Training Generalized Multi-Query Transformer ModelsSection 2 and Figure 2: query grouping and the MHA/GQA/MQA distinction.
4. Meta’s Llama 2 70B model cardContext, intended use, released sizes and custom license.