Qwen2.5

Designing the training corpus

A language model learns by predicting the next token over and over, so what it becomes good at depends on what the training text contains. Ten thousand near-identical product listings and ten thousand different worked problems have similar token counts but teach very different things.

The Qwen2.5 report describes four kinds of change. Earlier Qwen models scored candidate documents for quality, and low scorers were dropped. Mathematics and code data from the specialized Qwen2.5-Math and Qwen2.5-Coder projects were added. Synthetic examples written by larger models were filtered with a reward model before use. Finally, the team rebalanced subjects. Web text over-represents areas such as online shopping, social media and entertainment, so those were reduced, and under-represented areas such as science and technology were increased. [1] Filtering synthetic data reduces errors; it cannot guarantee that every generated answer is correct.

Picture a box of 100 training documents: 80 repetitive advertisements, 15 stories and 5 mathematical explanations. Doubling the box keeps the same imbalance. Replacing 30 of the advertisements with checked technical exercises changes what the model practices. The amount of data and its proportions are separate decisions, and both matter.

Trading model size for training data

A common rough estimate of the computation needed to train a dense transformer is 6ND floating-point operations, where N is the number of parameters and D the number of training tokens. At a fixed budget, halving N allows about twice as many tokens. The estimate covers the main matrix multiplications and leaves out the extra cost of attention on long sequences.

In the figure, start from a made-up 7-billion-parameter model trained on 1 trillion tokens. At 14 billion parameters, the same budget allows half as many tokens; at 3.5 billion, twice as many. The figure shows only the budget. A smaller model sees more data but has fewer weights with which to learn from it, and no rule says the two effects cancel out.

The case for training small models longer comes from deployment. Every generated token passes through the whole dense model, so a smaller model is cheaper for every request it ever answers. Paying more once, in training, can save money across millions of requests. That is why Qwen2.5 comes in seven sizes: users can pick one that fits their hardware and budget. The sizes do not share a license. Most use Apache 2.0, while the 3B and 72B models have Qwen’s own licenses. [2]

The memory a conversation needs

The weights stay fixed while the model runs, but each conversation adds to the KV cache: the stored keys and values of earlier tokens, which attention consults to retrieve context. That memory comes on top of the weights.

Qwen2.5-7B has 28 layers. In each one, 28 query heads share 4 key–value heads, so seven query heads use each stored key and value. This sharing, called grouped-query attention, comes from earlier work. The model card sets the default context at 32,768 tokens. Longer inputs, up to 131,072 tokens, need an extra setting called YaRN, which rescales the model’s position encoding, and not every runtime turns it on. [3]

Set the figure to 4,096 tokens. With each number stored in two bytes (bf16), the cache for one conversation is 2 (key and value) × 28 layers × 4,096 tokens × 4 heads × 128 numbers per head × 2 bytes, about 224 MiB. Double the context and the cache doubles. Switch to MHA, where every query head has its own keys and values, and the cache is seven times larger, about 1.5 GiB. The figure counts memory, not speed.

Comparing like with like

The instruction-tuned checkpoints add training after pretraining so the model responds helpfully to requests. The launch emphasized following instructions and producing structured output such as valid JSON. Separate Coder and Math models were tuned for programming and mathematics. Producing valid JSON and proving a theorem are different skills, and a single averaged leaderboard score hides the difference. [2]

Because so much of Qwen2.5’s improvement came from data and training rather than architecture, fair comparisons take care. Compare base models with base models when studying pretraining, and instruction-tuned models with each other when studying behavior. Keep the prompt format, context setting and test data the same. Otherwise a difference in score might come from any of those settings rather than from the model.

Simplifications and limits

• The training-budget figure approximates the main matrix work of a dense model. It ignores attention’s dependence on sequence length and is not a fitted scaling law.
• Qwen2.5-Turbo, which offered a one-million-token context, was available only through an API. That context length does not apply to the September open-weight models.

Follow the idea

Qwen3

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Qwen2.5 Technical ReportSections 2–4; corpus selection, mixture, long-context stages and architecture table.
2. Qwen2.5 launch, September 19, 2024Release date, dense model sizes, license exceptions and task emphasis.
3. Qwen2.5-7B-Instruct model cardModel dimensions, YaRN configuration and default context length.