Llama 3.1 · 8B, 70B, 405B

A conventional design at a larger scale

Meta released Llama 3.1 on July 23, 2024, in 8B, 70B and 405B sizes, with a 128,000-token context window and support for eight languages. The announcement highlighted another use for the largest model: generating training data to improve the smaller ones. [1]

All three are dense transformers: every token passes through every weight. Mixtral, by contrast, sends each token through only a few of its experts. In a dense model, the parameter count is a fair first guide to the computation each token needs.

Why keep a familiar design? The architecture is only one of the things that decide how good a model is. Which documents survive filtering, how often each subject appears, and which answers are used in later training can all change the model while its block diagram stays almost the same.

Spending a fixed training budget

Training cost is often estimated as about 6ND floating-point operations (FLOPs), where N is the number of parameters and D the number of training tokens. The 6 counts the arithmetic for the forward and backward passes, per parameter, per token. The estimate leaves out some costs, such as attention over long sequences. For Llama 3.1 405B, trained on 15.6 trillion tokens, 6ND gives about 3.8 × 10²⁵ FLOPs, which matches the figure the paper reports. [3]

With a fixed budget, more parameters means fewer tokens. In the figure, start at 7 billion parameters and 1 trillion tokens. Slide to 14 billion parameters and the same budget buys only half a trillion tokens. The curve shows the trade, not which point on it produces the best model.

The Chinchilla study measured where the best point lies for a given budget. It found that parameters and training tokens should grow together, at roughly 20 tokens per parameter. [2] That answer minimizes training cost for a given quality, but it ignores the cost of using the model. Training is paid once; every request to a deployed model costs computation again, and a smaller model costs less per request. Meta trained the 405B model at about the size its own scaling experiments called compute-optimal, then trained the 8B and 70B models on far more tokens than that rule suggests, paying more for training to get better small models. [3]

Choosing which tokens to train on

A token count says nothing about what the tokens contain. The Llama 3 report describes removing duplicate documents, filtering out low-quality text, running experiments to choose how much of each subject to include, and taking care to keep mathematics and code intact when extracting text from web pages. It also describes grouped-query attention at every size and a larger vocabulary of 128,000 tokens. [3] The gains come from the whole training recipe; no single new layer type explains them.

Picture two training sets of equal length. One repeats the same copied tutorial a hundred times. The other holds a hundred different explanations, each with a correct worked example. The token counts match, but the second teaches far more. Removing duplicates makes a fixed budget less repetitive, and careful extraction keeps an equation from being mangled into nonsense.

Filtering can also remove things worth keeping, such as dialects, difficult technical writing, or languages with little web text. The right test of a data change is which evaluation tasks improve and which get worse, not whether a generic quality score goes up.

Making 128,000 tokens usable

Raising the context limit in a configuration file does not teach a model to use long inputs. The report describes extending the context in six stages, from 8,000 tokens up to 128,000, spending about 800 billion training tokens on longer documents. Post-training then added examples whose answers depend on information far back in the input. [3] The released 3.1 models read and write text only; the paper’s image and speech experiments were not part of the release.

Consider a contract that defines a term on page one and adds an exception near the end. Fitting the whole contract into the context only shows that it fits. Answering a question correctly means finding the definition, finding the exception and applying both. Tests that hide one fact in a long document and ask for it back are useful, but passing them does not show that the model can reason reliably across a whole contract.

The 128,000 tokens are shared between the prompt and the reply, and the two take very different amounts of time. The model reads a prompt in parallel, many tokens at once, but writes a reply one token at a time. A 100,000-token prompt with a short answer and a short prompt with a 100,000-token answer fill the same context but take very different times.

Choosing between good answers

After pretraining, Llama 3.1 was trained on example conversations and then with direct preference optimization (DPO). Each DPO training example is a pair of answers to the same prompt, one marked preferred and one rejected. Training raises the probability of the preferred answer relative to the rejected one, measured against a frozen copy of the model from before this stage. The method comes from an earlier paper. [4]

Take the question “What is 6 × 7?” and two answers: “42” and “42, because six groups of seven make forty-two.” If the user is learning arithmetic, annotators may prefer the explanation. If the prompt says “reply with only the number”, they should prefer “42”. Which answer is better depends on the request, and the preference data has to capture that.

A model release is a bundle of decisions. When you read about one, ask separately how the model was pretrained, how its behavior was shaped, what it costs to run, and whether its evaluations resemble the task you care about.

Simplifications and limits

• The compute figure is a rough estimate, not a fitted scaling law or a hardware measurement.
• The paper’s image and speech research models were not released as Llama 3.1 weights.
• The 128,000-token window covers prompt and reply together, and fitting text into it does not guarantee that the model uses every detail.

Follow the idea

Llama 2 · 7B, 13B, 70BLlama 4 · Scout and Maverick

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Introducing Llama 3.1July 23 release, family sizes, 128K context, eight languages and synthetic-data applications.
2. Training Compute-Optimal Large Language ModelsSections 2–3: training-compute allocation and parameter/token tradeoffs; widget is an independent toy calculation.
3. The Llama 3 Herd of ModelsTable 1 and Sections 3.1–3.4, 4.1: Llama 3.1 data, architecture, context curriculum and post-training.
4. Direct Preference OptimizationSections 3–4: chosen/rejected pair objective relative to a reference policy.