Open-weight language models
The ideas that changed what a model can do, and what it costs to run
A language model can improve by learning from better examples, storing the past differently, activating fewer weights, or spending more computation before answering. These tutorials follow those mechanisms from a small worked example to the released model.
Start with a question below, or browse the release chronology. Each lesson names what changed, what was inherited, and what the evidence leaves uncertain.
Four routes through the ideas
- Before deployment: change the weights
- Pretraining learns from a corpus. Supervised fine-tuning imitates examples. Preference learning and reinforcement learning change which outputs receive credit.
- Inside a token: choose the work
- Attention retrieves context. A mixture of experts selects feed-forward networks. More stored parameters need not mean proportionally more arithmetic for each token.
- Across tokens: choose what to remember
- A KV cache preserves earlier keys and values. Latent attention compresses them. Recurrent layers update a fixed-size state; hybrid models combine approaches.
- Before the answer: choose more steps
- Reasoning, verification, tool calls, and parallel tasks spend extra inference compute. Their usefulness depends on training, the task, and the reliability of feedback.
Browse the tutorials
40 tutorials
Llama 2 · 7B, 13B, 70B
Meta released Llama 2 as both a base model and a chat model, described how the chat model was trained, and gave the 70B version a cheaper way to store a conversation.
Mistral 7B
Mistral 7B shared stored keys and values across attention heads and let each layer look back only 4,096 tokens, cutting the memory a long conversation needs.
Mamba
Mamba replaced the growing attention cache with a fixed-size memory whose updates depend on the current token, and found a fast way to train it on GPUs.
Mixtral 8×7B
In each layer, a router sends every token to two of eight feed-forward networks, so Mixtral stores 46.7 billion parameters but uses about 12.9 billion for each token.
Phi-3 · mini
Phi-3-mini trained a 3.8-billion-parameter model on heavily filtered web text and generated lessons, aiming for strong reasoning at a size that fits on a phone.
DeepSeek-V2
DeepSeek-V2 cached one small learned vector per token instead of full keys and values, and paired it with many small experts, cutting both memory and computation.
Gemma 2 · 9B and 27B
Gemma 2 trained its 9B model on a larger teacher’s next-token probabilities instead of only the observed text, and alternated local and global attention layers to limit memory.
Llama 3.1 · 8B, 70B, 405B
Llama 3.1 kept a conventional dense transformer and improved it through data curation, training scale, staged extension of the context to 128,000 tokens and preference training.
Qwen2.5
Qwen2.5 kept a familiar dense transformer and improved it mainly by growing and rebalancing its training data, and came in seven sizes for different budgets.
Tülu 3
Tülu 3 published a complete post-training recipe that ends with reinforcement learning on rewards from automatic checks, such as whether a math answer is correct.
DeepSeek-V3
DeepSeek-V3 scaled V2’s design to 671 billion parameters and trained it efficiently with bias-based expert balancing, 8-bit arithmetic and multi-token prediction.
DeepSeek-R1
DeepSeek-R1 used reinforcement learning on automatically checked answers to train long step-by-step reasoning, then taught smaller models to imitate its solutions.
LLaDA 8B
LLaDA generates text by filling in a reply made of blanks over several rounds instead of writing left to right, and shows why that is not automatically faster.
Gemma 3 · 1B, 4B, 12B, 27B
Gemma 3 made five of every six attention layers look only at the last 1,024 tokens, which kept the cache small enough for 128,000-token inputs and images.
Llama 4 · Scout and Maverick
Llama 4 was Meta’s first Llama release to use a mixture of experts and to train on images and text together from the start.
BitNet b1.58 2B4T
BitNet b1.58 2B4T trained a 2-billion-parameter model whose weights take only the values −1, 0 and +1, instead of rounding a finished model afterward.
Qwen3
Qwen3 put a switchable thinking mode into one model, trained it in four stages for the largest models, and distilled it into the smaller ones.
Phi-4-reasoning · reasoning-plus
Phi-4-reasoning fine-tuned the 14B Phi-4 on long worked solutions to carefully chosen problems; reasoning-plus added reinforcement learning on checked answers.
Falcon-H1 · 0.5B to 34B
Falcon-H1 runs attention and a Mamba-2 recurrent branch side by side inside each block and mixes their outputs, instead of alternating the two in separate layers.
Phi-4-mini-flash-reasoning
Phi-4-mini-flash-reasoning stores the full history once, shares it across layers, and lets some layers reuse earlier results through gates, which speeds up long answers.
Kimi K2
Kimi K2 trained a trillion-parameter mixture of experts stably with the MuonClip optimizer, and learned tool use from large numbers of generated practice sessions.
GLM-4.5
GLM-4.5 combined reasoning, coding and tool use in one sparse model, and kept its reinforcement learning effective by raising problem difficulty as the model improved.
gpt-oss-20b and gpt-oss-120b
OpenAI’s first open-weight language models since GPT-2 combine sparse experts, mixed local and full attention with an attention sink, four-bit expert weights and a structured format for tool calls.
Qwen3-Next
Qwen3-Next makes three of every four layers update a fixed-size memory instead of a growing cache, keeps full attention in the rest, and uses about 3 billion of its 80 billion parameters per token.
DeepSeek-V3.2 / Exp
DeepSeek-V3.2 uses a small, fast indexer to choose which earlier tokens each query reads, so the expensive attention step touches only a selected subset.
DeepSeek-OCR
DeepSeek-OCR compresses an image of a page into a few hundred vision tokens and decodes the text back out, testing whether images can hold text more compactly than text tokens.
Kimi K2 Thinking
Kimi K2 Thinking interleaves reasoning with hundreds of tool calls, and was trained with four-bit expert weights so that long tasks move less data through memory.
Olmo 3
Ai2 published Olmo 3’s data, code and checkpoints from every training stage, so researchers can change one step and measure its effect on the final model.
Kimi K2.5
Kimi K2.5 added image and video input to Kimi K2, and trained an orchestrator to split suitable tasks among helper agents working in parallel.
GLM-5
GLM-5 lets rollout generation and learning run on separate schedules so slow agent tasks do not stall training, and adopts sparse attention to cut the cost of long contexts.
Qwen3.5
Qwen3.5 trained images and text together on top of Qwen3-Next’s hybrid architecture, whose fixed-size memory layers keep long visual inputs affordable.
Phi-4-reasoning-vision-15B
Phi-4-reasoning-vision pairs a vision encoder that keeps fine detail with a language model trained to choose between answering directly and reasoning at length.
Leanstral / 1.5
Leanstral is trained to write formal Lean proofs inside real projects, using the proof checker’s feedback to guide its work and the checker’s verdict as its training reward.
Gemma 4 · E2B, E4B, 26B-A4B, 31B, later 12B
Gemma 4 mixes sparse and dense models, counts parameters in several ways, adds a 12B model without separate image and audio encoders, and offers quantization-aware checkpoints, all under Apache 2.0.
DeepSeek-V4
DeepSeek-V4 merges groups of tokens into compressed cache entries, selects among them for distant context and keeps a local path for recent detail, making million-token contexts cheaper.
Kimi K3
Kimi K3 pairs a recurrent memory that forgets different features at different rates with attention over earlier layers’ outputs, in a 2.8-trillion-parameter mixture of experts.
Mamba-3 · MIMO 1.5B
Mamba-3 keeps a fixed-size recurrent state but updates it more richly, blending neighboring inputs, rotating the state, and doing more arithmetic for each memory access.
GLM-5.3-Flash
GLM-5.3-Flash is a new multimodal base model that mixes fixed-size memory layers with sparse attention and constrains how its four residual streams mix between layers.
Qwen3.8-Flash-Next
Qwen3.8-Flash-Next adds large n-gram lookup tables kept in host memory, a block-based sparse attention and gated residual streams to a Qwen3-Next-style hybrid.
DeepSeek-V4.1-Flash
DeepSeek-V4.1-Flash splits the model into an encoder that prepares input once and a decoder that generates, shares cached entries across layers and stores them in four bits, cutting the cost of input-heavy agent work.
No tutorials match those filters. Try another term or clear the filters.
A small vocabulary
- Token
- A unit the model processes: often a word fragment, sometimes a special control symbol or an encoded piece of an image.
- Parameter / weight
- A learned number. Billions of these numbers implement transformations; they are not a database of separately stored facts.
- Attention
- A learned comparison between a query and earlier keys, used to mix the associated values into the current representation.
- KV cache
- The keys and values retained from preceding tokens, so generation need not recompute them. This is distinct from model-weight memory.
- Expert
- A feed-forward network selected by a router inside a layer. It need not correspond to a human-readable specialty such as medicine or mathematics.
- Post-training
- Training after the base next-token or denoising objective, to shape instruction following, preferences, reasoning, and interaction.
- Distillation
- Training a student using a teacher's outputs or distributions. The details differ: imitation of full solutions is not identical to matching token probabilities.
- Open weights
- Downloadable learned parameters under a stated license. Training data, training code, and unrestricted reuse do not automatically follow.
Coverage and evidence
This is a selective history of open-weight language models, from Llama 2 in July 2023 to releases in September 2026, researched through October 3, 2026. Each page covers a release that introduced or made prominent a distinct technique. The collection does not try to list every checkpoint, fine-tune, quantization or unpublished training trick.
Each page covers one named release, or several sizes of it when they share the technique being explained. A page is dated by the day its weights became publicly available, where that can be verified, and explains any difference from the announcement or paper date. A later release gets its own page only when it adds a distinct, documented technique.
Every tutorial links to primary sources beside model-specific claims and lists the relevant paper sections. Reported results remain attributed to their authors. The figures calculate transparent teaching examples; they do not run these models, predict accuracy, or establish benchmark rankings.
License descriptions identify the published weight terms and are not a permission grant. Some families use custom licenses. The lesson distinguishes what can be downloaded from what can be independently reproduced.