DeepSeek-V3.2 / Exp

A cheap first pass decides which earlier tokens get expensive attention

DeepSeek-V2 made each cached token smaller, but each new token still has to compare itself with every earlier one. In a 100,000-token conversation, that is 100,000 expensive comparisons for every token generated. DeepSeek-V3.2 adds a cheap first pass. A small, fast scorer called the lightning indexer rates every earlier token, and the full attention calculation reads only the highest-rated ones. The design first shipped in DeepSeek-V3.2-Exp in September 2025. [1]

5 sections · about 8 minutes · Sparse compute

Release: 2025-09-29. DeepSeek Sparse Attention first shipped in the V3.2-Exp weights on September 29, 2025. The full V3.2 and V3.2-Speciale followed on December 1.Model: V3-family MoE backbone with a sparse-attention indexerWeight license: MIT

A smaller record still leaves a long search

Suppose an assistant has read a long investigation and is asked, “Which experiment used the copper coil?” The relevant sentence may be far from the end. A short sliding window would miss it. Full attention can check every earlier position, but its cost grows with the length of the history.

DeepSeek Sparse Attention (DSA) puts a lightweight scoring step, the lightning indexer, in front of the main attention. The indexer gives every earlier token a relevance score for the current query, and the main attention then reads the cached entries of only the highest-scoring tokens. Selection depends on content: a token is chosen because it matches the query, not because it is nearby. [2]

It works like a book’s index, which points to a few pages worth reading closely. The index still has to be consulted, though, and a learned index can point to the wrong pages. DSA makes the expensive step sparse; it does not make the whole process free.

Different questions select different tokens

In the figure, sixteen earlier tokens have made-up scores for the current query. Choosing the top four leaves twelve out of the main attention calculation. Switch to Query B or Query C: the scores change and different positions are selected. Nothing requires the selected positions to be recent or next to each other.

Now raise the number of tokens selected, called k. More of the history reaches the expensive attention step. Lowering k saves work but risks leaving out something useful. The slider shows no accuracy, because the scores are made up and no model is running.

In the released model, each query selects up to 2,048 earlier positions; the figure scales that down so every choice is visible. The cached entries are still MLA’s compressed vectors, inherited from earlier DeepSeek models. DSA changes which entries are read. That is a different kind of selection from expert routing, which chooses weights rather than tokens. [2]

The indexer scores everything; attention reads the chosen few

Simplified DSA selection with fixed, made-up scores. The indexer still scores every candidate, and the selection is not taken from the real model. Selecting by content can keep distant tokens that a fixed local window would drop.

Training the index before relying on it

V3.2 was made by continuing to train V3.1-Terminus, an existing DeepSeek model. In the first stage, the main model is frozen and keeps using full attention while only the new indexer trains, learning to predict where full attention puts its weight. In the second stage, the model switches to sparse selection, and the main model is unfrozen so it can adapt to reading only the selected tokens. The indexer and the main model are trained with separate objectives. [2]

Why this order? Imagine a new librarian whose index points to random pages. Taking away access to the full book on day one would leave both the index and the reader unreliable. The warm-up gives the index a sensible starting point before the rest of the model depends on it.

A made-up cost example shows why the design pays off. Take 1,000 earlier tokens, an indexer that costs 1 unit per token, and attention that costs 10 units per token it reads. Reading k selected tokens costs 1,000 + 10k units. At k = 100 that is 2,000 units, against 10,000 for full attention over all 1,000. The numbers are invented; they show the shape of the saving, not a measured speed-up.

From full reading to selective reading

Simplified continued-training sequence. Stage lengths and settings are omitted. A trained model needs time to adapt when its access to earlier tokens changes.

Cheaper reading makes long tasks practical

The full V3.2, released in December 2025, adds substantial post-training for reasoning and tool use. The model card describes synthetic agent tasks and environments used to train longer sequences of actions. A separate variant, V3.2-Speciale, focuses on reasoning, and the card states that it does not support tool calls. [3]

Consider an assistant that opens files, runs a test, reads the failure, edits a function and reruns the test. Every tool result adds text to the conversation. Cheaper attention lowers the cost of rereading that growing history at each step, but training still has to teach the assistant which action to take next.

Reading cheaply does not give a model good tool habits, and a good training curriculum does not reduce cache and attention costs. The two address different parts of a long task and should be evaluated separately.

Two meanings of “sparse”

In a mixture of experts, sparsity selects weights: which feed-forward experts process this token? In DSA, sparsity selects history: which earlier tokens inform this query? A model can use both, as V3.2 does, and the word “sparse” alone does not tell you which resource is being saved.

Reading fewer tokens does not shrink the stored cache by the same fraction. A later query can choose a different set, so the model generally has to keep all the candidates. DSA reduces the expensive reading step. DeepSeek-V4 goes further and compresses the stored history itself.

The official change log dates V3.2-Exp to September 29 and the full V3.2 to December 1. The two share the DSA architecture, but only the full release added the reasoning and agent training described above. [4]

Simplifications and limits

The selection figure uses made-up scores and does not run a model.
The lightning indexer still does work for every earlier token, so total cost still grows with length; only the expensive attention step becomes sparse.
V3.2-Exp and the full V3.2 share the DSA architecture but differ in post-training.

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

Official DeepSeek-V3.2-Exp model cardFirst public DSA model and inherited architecture.
DeepSeek-V3.2 technical reportSection 2: indexer, top-k selection and dense-to-sparse continued training.
Official DeepSeek-V3.2 model cardFull release, agent-task synthesis, Speciale limitations and MIT license.
DeepSeek official change logSeptember 29 Exp and December 1 full V3.2 release entries.