Kimi K3

Two directions of information flow

Picture a grid in which each column is a token in the sequence and each row is a layer. Attention across tokens moves information along a row: which earlier words matter to this one? Mixing across layers moves information up a column: which earlier processing of this same token is still useful? Both can be weighted combinations, but they solve different problems.

The K3 report specifies 93 layers: 69 Kimi Delta Attention (KDA) layers and 24 gated multi-head latent attention (MLA) layers, arranged as 23 groups of three KDA layers followed by one MLA layer, plus one final MLA layer. [6] MLA, introduced with DeepSeek-V2, stores a compressed vector for each earlier token, so its cache still grows with the context. Of the model’s 2.8 trillion parameters, only about 104 billion are active per token, because most of them sit in experts that each token uses only a few of. [1] Moonshot announced K3 on July 16, but the weights appeared on July 27, which is the date this page uses. [5] [2]

K3 therefore keeps two routes through the token history: a recurrent route, through the fixed-size state of the KDA layers, and an attention route, through the MLA layers’ cache, which can retrieve specific earlier tokens. The total parameter count says little about the memory cost of either route.

Forgetting different features at different rates

Kimi Delta Attention (KDA) builds on Gated DeltaNet, the recurrent layer described on the Qwen3-Next page. In Gated DeltaNet, one decay factor shrinks the whole state at each step, and a delta-rule update then corrects the association picked out by the current key. KDA replaces the single decay factor with a separate one for each channel, meaning each coordinate of the internal representation. A channel is a learned numerical feature, not a labeled concept. KDA first appeared in Moonshot’s earlier Kimi Linear model, together with a chunked algorithm that processes blocks of tokens at once without changing the result. [3]

As an analogy, imagine following a meeting: who is speaking now, and when the project is due. When the speaker changes, one part of your memory should update at once while the other should persist. A single decay factor weakens everything together. Separate factors let different channels hold information for different lengths of time. The analogy shows why selective forgetting helps; it does not mean any trained channel literally stores a deadline.

K3 adds one specific numerical change. Its retention factor, the fraction of each channel kept at each step, cannot fall below e⁻⁵, about 0.0067; before, it could approach zero. The fast chunked algorithm divides by products of these factors across tiles of 16 tokens, and without a floor such a product can become so small that the division overflows the number format. The floor makes it safe to run this part of the calculation as matrix multiplications on the GPU’s fast matrix units. It does not make memory permanent, because factors below one still shrink the state over many steps. [6]

The figure shows why a recurrent layer is attractive: its fixed-size state does not grow with each token. A fixed-size state must compress, though, and K3’s MLA layers still keep a growing cache. The model as a whole does not run in constant memory.

Letting later layers choose earlier outputs

A standard residual connection adds each layer’s output to the representation flowing up through the network, so the contributions of all earlier layers pile up with fixed, equal weight. Attention Residuals replace that fixed sum with attention over depth: each layer computes learned, input-dependent weights over the outputs of earlier layers. Keeping every earlier layer’s output costs memory and communication, so the original paper also proposes a block version, which K3 uses. This Block AttnRes sums contributions within each block of layers as usual, then attends over the finished block summaries, the current unfinished block and the original token embedding. The report groups K3’s 93 layers into eight blocks, the last of them partial. [4] [6]

Here is the difference with numbers. Suppose three earlier representations are 2, 5 and −1. A standard residual adds them: 6. Attention with weights 0.2, 0.7 and 0.1, which sum to one, gives 0.2 × 2 + 0.7 × 5 + 0.1 × (−1) = 3.8. The attention version can emphasize one source and play down another, which no fixed sum or overall rescaling can do.

The weights depend on the input even though the receiving layer’s query is a fixed learned vector, because the earlier representations it scores change from token to token. This is attention along depth, and it does not replace attention between tokens. A layer may need both: find the relevant earlier words, and choose which stage of processing holds the features it needs next.

Experts and low precision

K3’s experts use a design the report calls Stable LatentMoE. Each routed expert works on a 3,584-number projection of the 7,168-number representation that flows through the model, half its width, while the shared experts work at full width. Normalization and limits on internal values keep the numbers from growing before the experts’ result is added back. [6] The model card lists 896 routed experts, of which each token uses 16, plus 2 shared experts. [1]

K3 was also trained for low-precision deployment. The card describes quantization-aware training from supervised fine-tuning onward, with weights in MXFP4 and activations in MXFP8: four-bit and eight-bit floating-point formats in which each small block of values shares a scale factor. Those scale factors, and the parts kept at higher precision, add storage beyond the nominal four or eight bits. Each of these choices addresses a different cost, so they should not be collapsed into a single efficiency number. [1]

A fair test of any one mechanism changes only that mechanism while holding the data, compute and evaluation fixed. A release changes many things at once, so the component papers give more targeted evidence than a flagship benchmark table. What the released model shows is that the parts can work together in one system.

K3 separates three questions about information flow: what should persist through time, what should stay retrievable, and which earlier layer a later layer should draw on. Each has its own memory and computation cost.

Simplifications and limits

• KDA and Attention Residuals come from earlier component papers; the K3 report specifies the changes and settings used in this release.
• The weighted-sum example isolates mixing across layers; it is not a K3 layer.

Follow the idea

Qwen3-NextQwen3.8-Flash-Next

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Kimi-K3 official model cardLayer composition, expert counts, active/total parameters, quantization and custom license.
2. Kimi-K3 weight historyInitial public weight commit July 27, 2026, following July 16 product announcement.
3. Kimi Linear component paperKDA formulation, fine-grained gating and chunkwise execution; component predates K3.
4. Attention Residuals component paperSections 2–3; softmax aggregation along depth and block-level approximation.
5. Kimi K3 official announcementJuly announcement explicitly deferred public weights until July 27.
6. Kimi K3 full technical reportRead §§2.1–2.3, Table 1 and §3: bounded KDA decay, gated MLA, the deployed Block AttnRes design, and Stable LatentMoE. PDF retrieved from the repository’s raw-file endpoint.