Open-weight LLMsMistral 7B

Mistral 7B

Shrinking a conversation’s memory by sharing it across heads and limiting how far back each layer looks

A model’s weights take a fixed amount of memory. The record it keeps of the conversation so far does not: it grows with every token. Mistral 7B, released by the French company Mistral AI in September 2023, slowed that growth with two changes to attention. Its 32 query heads share 8 sets of stored keys and values, and each layer looks back over only the most recent 4,096 tokens. Neither idea was new. The release showed them working together in a small model that anyone could download. [1]

5 sections · about 8 minutes · Memory

Release: . Mistral AI released v0.1 on September 27, 2023, and published the technical paper in October. This page describes v0.1.Model: 7B-class dense modelWeight license: Apache 2.0

What the KV cache stores

Take the sentence “Maya put the red notebook beside the blue one, then picked up the red …”. To predict the next word, the model has to find “notebook” earlier in the sentence. Attention does this. The current token produces a query, a list of numbers describing what it needs. Each earlier token offers a key, which is compared with the query, and a value, the information it passes on when the comparison matches.

The keys and values of earlier tokens never change, so the model stores them instead of recomputing them for every new token. This store is the KV cache. Each new token adds another key and value in every layer. A server shares one copy of the weights among all its conversations, but each conversation needs its own cache.

Two questions set the size of the cache. How many keys and values must the model keep for each token? And how far back must each layer be able to look? Mistral 7B changed the answer to both.

Sharing keys and values across heads

Attention runs as several heads at once, each with its own query, so different heads can look for different things. In standard multi-head attention, each head also has its own keys and values. Multi-query attention (MQA) goes to the other extreme: every head shares a single key and value. Grouped-query attention (GQA) sits between the two. Every head keeps its own query, and each group of heads shares one key and one value. [2]

Mistral 7B has 32 query heads and 8 key–value heads, so each stored key and value serves four query heads. [3] Those four heads can still look for different things, because their queries differ. They search the same stored records.

The figure uses Mistral 7B’s real layer and head dimensions. Doubling the number of tokens doubles the cache. Switching from 32 key–value heads to 8 cuts it to a quarter at any length. The figure ignores the sliding window described in the next section, so that the effect of sharing can be seen on its own.

Two separate multipliers: length and key–value heads

Simplified KV-cache size for one conversation, with each number stored in two bytes (bf16), using Mistral 7B v0.1’s layer and head dimensions. It ignores the sliding window, the weights, batching and other runtime memory. Sharing shrinks the memory for each token. It does not limit how many tokens are kept.

Looking back only 4,096 tokens

Mistral 7B v0.1 also uses sliding-window attention. In each layer, a token attends only to the 4,096 tokens before it, not to the whole conversation. Once a token falls outside the window, its key and value are no longer needed, so the cache can overwrite that slot with a new token’s. The paper calls this a rolling buffer: the cache stops growing once it holds 4,096 positions. [3] Later Mistral models, including Mixtral, do not all use this design.

The figure shrinks the idea to 16 positions and a window of 4. Pick a query position near the end and switch between Full and Local. With full attention, the number of visible earlier positions keeps growing. With local attention, a band of fixed width slides along. The blank cells are comparisons this layer skips.

Information from before the window is not simply lost. In the first layer, a token sees the 4,096 tokens before it. Each of those tokens has already gathered information from its own window, so by the second layer the reach is roughly twice as far, and it keeps extending up the stack. The paper puts the theoretical reach after 32 layers at about 131,000 tokens (32 × 4,096). As with a message passed along a line of people, though, details can be lost at each step. A long theoretical reach does not guarantee that the model can recall a distant detail. [3]

Watch the window move

Simplified attention pattern for one layer, with 16 positions and a window of 4 standing in for Mistral’s 4,096. The “Toy hybrid” option is a generic teaching pattern, not Mistral’s design. Each layer looks only nearby. Older information arrives indirectly, through earlier layers.

The two savings multiply

Take a conversation of 32,768 tokens. Without a window, every layer stores keys and values for all 32,768. With a 4,096-token window, it stores at most 4,096: eight times fewer. Sharing 8 key–value heads among 32 query heads cuts storage by another factor of four. Together, the cache holds 8 × 4 = 32 times fewer numbers than full attention with separate keys and values for every head. This is arithmetic about the cache alone. It does not tell you how much faster the model runs or how much total memory a server saves.

The two changes remove different kinds of duplication. Grouped-query attention shares records across heads. The window limits how far back in time each layer reaches. Restricting attention to nearby positions was not new either: the Sparse Transformer used similar patterns in 2019. Mistral’s contribution was a practical combination of the two in a small, capable model. [4]

Whether the model can still use distant information is a question for testing. A useful test asks about a specific detail far back in a long document and checks the answer. The paper reports results on its own benchmarks; those results cannot show that the savings cost nothing on every task.

What the release made possible

Mistral AI published the base model and an instruction-tuned version under Apache 2.0, a standard open-source license that permits commercial use and modification with few conditions. Developers could download, inspect, fine-tune and host the model without negotiating terms. [1]

When you compare models, track three numbers separately: the memory for the weights, the memory for each conversation’s cache, and the computation for each generated token. A design change can improve one and leave the others almost untouched. The next pages take different routes. Mixtral reduces the computation per token, and DeepSeek-V2 shrinks the cache.

Simplifications and limits

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

  1. Mistral: Mistral 7B releaseSeptember 27 release, public weights and Apache 2.0.
  2. GQA: Training Generalized Multi-Query Transformer ModelsGQA mechanism and its position between MHA and MQA.
  3. Mistral 7B technical reportSection 2, Table 1, sliding windows and rolling buffer cache.
  4. Generating Long Sequences with Sparse TransformersPrior sparse-attention work; establishes that attention locality predates this release.