DeepSeek-V4

Compressing features versus compressing time

Picture the cache as a table, with one row per token and one column per stored number. DeepSeek’s earlier latent attention compressed the columns, storing fewer numbers per token. V4 also compresses the rows, merging groups of neighboring tokens into one entry. The two are separate axes of the same storage problem.

A made-up table with 1,024 rows and 256 columns holds 262,144 numbers. Cutting the width to 64 columns leaves 65,536. Keeping the width but merging every eight rows into one leaves 32,768. Doing both leaves 8,192, which is 32 times fewer than at the start. Because the two axes are independent, their savings multiply.

The cost is detail. Merging rows can blur individual tokens, so the merging is learned, and the model is trained to keep what it needs in the merged entries. A smaller cache alone does not show that recall survives.

Compress, then select

Compressed Sparse Attention (CSA) first merges the sequence into fewer cached entries. A learned indexer, like the one in DeepSeek-V3.2, then picks which merged entries each query reads. Alongside this, a local sliding-window path gives full detail on the most recent tokens. [2]

Imagine 64 earlier passages compressed into 16 entries, of which the query reads 4. The figure starts at the selection step: each bar stands for one compressed entry, not one original token. Switch queries to see distant entries chosen, and change the number selected to change how much compressed history is read directly.

Reading four compressed entries can bring in information from many more than four original tokens, but it does not recover them exactly. The compression is learned, and it loses detail.

A coarse overview alongside local detail

V4 alternates CSA layers with Heavily Compressed Attention (HCA) layers. HCA merges the sequence much more aggressively and then attends to every one of the resulting entries. “Every” refers to the compressed entries, not the original tokens. Both layer types also keep the local attention path. [2]

A map analogy separates the roles. A coarse map shows the whole region cheaply, like HCA. An index lets you zoom in on a few relevant areas, like CSA. Street view keeps your immediate surroundings sharp, like the local path. None of them alone gives every detail at every distance.

In a made-up example, 4,096 original tokens compressed 16 to 1 become 256 entries. Reading all 256 is full attention over the coarse version, yet it is sixteen times cheaper than reading all 4,096 tokens. This shows the counting only; it is not V4’s real compression ratio.

Keeping the residual streams stable

V4 also adopts manifold-constrained hyper-connections (mHC). In a standard transformer, a residual connection carries each layer’s input forward and adds the layer’s output to it, so information passes up the network along one stream. Hyper-connections widen this into several parallel streams that mix with each other between layers. mHC constrains that mixing so the streams cannot amplify each other without limit over many layers. [3]

A small example shows the danger and the fix. Multiplying by 2 again and again gives 1, 2, 4, 8 and keeps growing. Mixing with a matrix whose entries are nonnegative and whose rows and columns each sum to one behaves differently: each output is a weighted average of the inputs. With the matrix [[0.8, 0.2], [0.2, 0.8]], the inputs [2, 6] become [2.8, 5.2], and repeated mixing keeps the values between 2 and 6.

The constraint applies to how the residual streams mix, not to every operation in the network. It does not guarantee that no activation anywhere in the model can grow.

What the release contains

The April release included two sizes: V4-Pro, with 1.6 trillion parameters in total and 49 billion active per token, and V4-Flash, with 284 billion in total and 13 billion active. Both are available under the MIT license. A small active fraction does not make either of them small to store. [1] [4]

V4 keeps the mixture of experts and multi-token prediction of earlier DeepSeek models and adopts the Muon optimizer, described on the Kimi K2 page. These are separate choices, so improvements in quality cannot all be credited to the attention compression described here. [4]

A long supported context says how much the model accepts, not whether it will find a fact buried deep inside. To judge a long task, ask which details must survive compression, how much new input arrives each turn, and whether the serving system can reuse earlier work. That last question leads to DeepSeek-V4.1-Flash, which reuses caches across layers.

Simplifications and limits

• Token counts and compression ratios in the examples are made up; they are not V4’s settings.
• The selection figure starts after compression and omits the local path.
• The June report describes the architecture of the April preview; later checkpoint updates are not treated as new designs.

Follow the idea

DeepSeek-V3.2 / ExpDeepSeek-V4.1-Flash

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. DeepSeek-V4 official preview releaseApril 24 date, open-weight links, Pro/Flash sizes and million-token context.
2. DeepSeek-V4 technical reportSections 2.1–2.3: sequence compression, CSA, HCA and local attention.
3. mHC: Manifold-Constrained Hyper-ConnectionsResidual mixing and doubly stochastic constraints.
4. Official DeepSeek-V4-Flash model cardArchitecture summary, inherited components, download availability and MIT license.