Qwen3-Next
How a running memory and a searchable history cooperate
A model reading a long conversation can keep every detail available to look up, which costs memory that grows with the conversation, or it can fold the past into a fixed-size summary, which is cheap but loses detail. Qwen3-Next, released by Alibaba’s Qwen team in September 2025, does both. Three of every four layers keep a running summary, and the fourth uses ordinary attention over the full history. The model also uses only about 3 billion of its 80 billion parameters for each token. The model card reports that it beats the dense Qwen3-32B base model at about a tenth of the training cost, with ten times the throughput on inputs over 32,000 tokens. [2]
Release: . Qwen released the model publicly on September 11, 2025. The repository shows the weights uploaded on September 9; the page uses the public release date.Model: 80B total · 3B activeWeight license: Apache 2.0
Two kinds of remembering
A notebook keeps every line so you can look it up later. A running total keeps a conclusion and throws away the list that produced it. Both are memory, but they answer different questions. From a total of 27, you cannot tell whether the last two entries were 10 and 17 or 8 and 19.
Attention with a KV cache works like the notebook: it stores keys and values for every earlier token. A recurrent layer works like the running total: each new token updates a fixed-size state. Qwen3-Next’s recurrent layers use Gated DeltaNet, a design from earlier research. Its state is a matrix that stores associations between keys and values; it can forget, and it can revise a specific association. The state’s size is set by the architecture, not by the length of the conversation. [1]
The figure compares a made-up state of 256 numbers with a cache that grows by 64 numbers per token. At 128 tokens the cache holds 8,192 numbers, and the state still holds 256. The sizes are kept small for teaching. Holding the same count of numbers would not mean holding the same information.
A fixed state beside a growing history
Correcting a memory instead of adding to it
The delta rule updates a stored association in proportion to its error. Take a stored value of 6 and a new target of 10. With a write strength of one-half, the update is 6 + 0.5 × (10 − 6) = 8. Seeing the same target again gives 9, then 9.5: the memory converges on 10. Simply adding 10 each time would instead make the stored value grow without limit.
Gated DeltaNet first shrinks the old state by a learned decay factor and then applies the correction. With a decay factor of 0.5, the 6 first becomes 3, and the same half-strength correction toward 10 gives 3 + 0.5 × (10 − 3) = 6.5. The decay decides how much of the existing memory to keep; the delta update decides how to revise the association picked out by the current key. In the real layer, the state is a matrix and different keys can overlap, so a fact does not sit in one slot. The single number here only shows the mechanism. [1]
The update, with one number
Keeping a direct route to the past
The released 80B-A3B model has 48 layers in twelve repeating groups: three Gated DeltaNet layers followed by one gated-attention layer, which is full attention whose output passes through a learned gate. It also uses experts: 512 routed experts per layer, of which each token uses 10, plus 1 shared expert. These are separate savings. The recurrent layers reduce the work of attending over the history, and the experts reduce the feed-forward weights each token uses. [2]
Think of a long meeting transcript. A compact state can track the current plan and who keeps speaking. A full-attention layer can still look up an unusual identifier mentioned at the start. Access does not guarantee that the model reproduces the identifier correctly, and the trained layers need not divide the work along these human lines. The example only shows why combining the two is useful.
The full-attention layers still keep a cache that grows with the conversation, so the hybrid as a whole is not constant-memory. It also differs from sliding-window attention. A sliding window keeps exact recent tokens and drops everything older; Gated DeltaNet keeps a compressed summary of everything it has read.
Speed depends on the serving software
Qwen3-Next also includes multi-token prediction, which can propose draft tokens for speculative decoding: the main model checks several proposed tokens at once and keeps the ones it agrees with. The model card warns that real throughput depends heavily on the serving software, and that multi-token prediction is not generally available in the basic Hugging Face Transformers implementation. [2]
Measuring speed therefore involves several settings at once: context length, batch size, the attention implementation, where the experts are placed, and how often drafts are accepted. A small number of active parameters does not guarantee a correspondingly short response time, because the hardware still stores all the experts, moves data between them and coordinates the work.
The model accepts 262,144 tokens natively, and a separately documented recipe extends that further. This page covers the Instruct model, which answers directly; a companion Thinking model reasons step by step first. Mixing them up changes both response time and the form of the output. [2]
Simplifications and limits
- The memory figure shows how the two kinds of memory scale. It does not implement or estimate the released model’s memory.
- Gated DeltaNet came from earlier research; Qwen3-Next’s contribution is building it into a large hybrid model.
Sources and further reading
Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.
- Gated Delta Networks: Improving Mamba2 with Delta RuleSections 2.3–3.1; state, delta correction and learned decay. Component work predates Qwen3-Next.
- Qwen3-Next-80B-A3B-Instruct model cardArchitecture schedule, expert counts, context, checkpoint modes and implementation caveats.
- Official Qwen3-Next model historyWeights staged September 9; release metadata and license finalized September 11, 2025.