DeepSeek-V2

Storing a compact summary instead of full keys and values

Suppose a device records four readings: x, 2x, 3x and 4x. Storing all four wastes space. Store x and the rule, and you can rebuild the rest. Real keys and values are not this tidy, but they contain a lot of redundancy, and a model can learn a smaller set of numbers from which they can be rebuilt.

That smaller set is a latent vector: a short list of coordinates in a compact internal space that the model learns. Multi-head latent attention (MLA) is built on it. For each token, a learned projection squeezes the token’s internal representation into a short latent vector, and the cache stores only that vector. When attention needs keys and values, other learned projections produce them from the latent. All these projections are trained with the rest of the model, so the model learns to put into the latent what attention will need. [2]

This is a different saving from quantization. Quantization stores each number in fewer bits. MLA stores fewer numbers. The two can be combined, and either one loses information unless training adapts to it.

How much smaller the cache gets

The figure starts from a made-up example: keys and values that would take 512 numbers per token per layer, compressed into a 64-number latent. At two bytes per number, that is 1,024 bytes versus 128. Multiply both by the number of tokens and the number of layers, and the ratio stays at one-eighth. Drag the latent width up and the saving shrinks, because less is being removed. The figure counts storage only; it cannot show how much accuracy a narrower latent would cost.

The real model saves far more. DeepSeek-V2 has 128 attention heads, each with keys and values of 128 numbers. Standard multi-head attention would cache 2 × 128 × 128 = 32,768 numbers per token per layer. MLA caches a 512-number latent plus a 64-number key that carries position: 576 numbers, about 57 times fewer. The paper reports that V2’s cache is 93.3% smaller than that of DeepSeek’s earlier 67-billion-parameter dense model. [2] The next section explains the separate 64-number position key.

Why position needs separate handling

MLA has a further trick: it does not need to rebuild past keys at all. Attention scores a query q against a key k with a dot product, qᵀk. If the key is rebuilt from the latent c by a learned matrix U, so that k = Uc, then qᵀk = qᵀUc = (Uᵀq)ᵀc. The model can multiply the new query by Uᵀ once and compare the result directly with every cached latent. The full keys for past tokens never have to be written out. The algebra is exact. [2]

Position breaks the trick. The model must tell “dog bites man” from “man bites dog”, so attention needs to know where each token sits. DeepSeek-V2, like most current models, uses rotary position embeddings (RoPE), which rotate parts of each query and key by an angle that depends on the token’s position. A rotation that differs for every past token cannot be folded into the single matrix Uᵀ. DeepSeek-V2 therefore splits each key in two: a content part rebuilt from the latent, where the trick works, and a small separate part of 64 numbers per token that carries the rotary position and is cached directly. [2]

The latent is not a compressed copy of some full-size model’s cache. It is a bottleneck built into the architecture from the start of training, so the model learns to work through it.

Shared experts plus many small routed ones

The feed-forward layers use DeepSeekMoE, an expert design from an earlier DeepSeek paper. [3] It makes two changes to a Mixtral-style mixture of experts. First, it splits the experts into many smaller ones, so each token can combine more of them. Second, it adds shared experts that process every token. The paper’s reasoning is that some knowledge is needed by almost every token; if that knowledge lives in shared experts, the routed experts do not each need their own copy. Each DeepSeek-V2 layer has 2 shared experts and 160 routed experts, of which the router picks 6 for each token. [2]

The figure scales this down to 8 routed experts and 1 shared expert. Change how many routed experts each token uses, and the shared expert stays active throughout. “2 of 8 experts” therefore understates the work per token, because the shared expert and attention also run on every token.

Whether the shared experts really hold common knowledge is the design’s motivation; the paper does not prove it. The figure shows only how the computation is organized.

Two savings for two different costs

The repository lists a 128,000-token context window and pretraining on 8.1 trillion tokens. [1] A long context window means the model accepts long inputs; it does not guarantee that the model uses every part of them well. A compact cache also does nothing to shrink the 236 billion parameters, all of which must stay in memory.

The two savings help with different workloads. Double the number of conversations a server handles, and the weights stay the same while the total cache roughly doubles; MLA matters most there. Add more experts, and the cache stays the same while the weights grow. MLA reduces the memory left behind by earlier tokens, and the expert design limits the computation spent on the current one. DeepSeek-V3 kept both and focused on training them efficiently at a much larger scale.

Simplifications and limits

• The latent-width figure does not model quality, the separate position key or the exact MLA computation.
• The expert figure uses scaled-down counts and cannot predict speed.
• The weights use DeepSeek’s own model license; the code uses the MIT license.

Follow the idea

Mistral 7BMixtral 8×7BDeepSeek-V3

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Official DeepSeek-V2 repositoryRelease news, specifications, downloads and license distinction.
2. DeepSeek-V2 technical reportSections 2.1.2–2.1.4: joint KV compression, projection absorption and decoupled RoPE.
3. DeepSeekMoE technical reportFine-grained expert segmentation and shared expert isolation.