Qwen3.8-Flash-Next

Four counts that one parameter number hides

The model card lists a 125-billion-parameter backbone with about 6 billion active per token, plus 51 billion parameters in n-gram embedding tables and a 4-billion-parameter multi-token-prediction module. These must be kept apart. Quoting only 125 billion omits a large stored table; adding everything together and calling it active would be just as misleading. [1]

The model inherits Qwen3-Next’s mix of Gated DeltaNet and attention layers, but replaces the attention layers with Qwen Sparse Attention (QSA). The architecture report, published on August 31, five days after the weights, also introduces a Gated Residual and the offloaded n-gram tables. [2]

For any model, four questions are worth asking separately: how many numbers are stored, how many are fetched for one token, how much arithmetic is done, and which device holds each part. No single parameter count answers all four.

Lookup tables for short token patterns

An n-gram is a short run of consecutive tokens: two for a bigram, three for a trigram. The model hashes each such run to a row in a learned embedding table and adds that row’s vector to its computation. The rows are trained vectors, not saved passages of text. The report keeps the tables in the host computer’s main memory and adds their output at the second layer, so the transfer can happen while the first layer is computing. A lookup adds stored knowledge at a different cost from an expert network: it is read, not computed. [2]

Imagine a table with a million rows of 16 numbers each. Looking up one row reads 16 numbers, not 16 million. The table still takes up space, though, and if a row must cross a slow connection and cannot be fetched early, the lookup can stall the computation.

That explains both the opportunity and the limit. Which row to fetch depends only on the recent tokens, which are already known, so it can be fetched early. But a table cannot reason: recognizing a familiar phrase is different from resolving a new contradiction. The backbone still has to interpret and use what the table returns.

Selecting small blocks before expensive attention

QSA’s lightweight indexer first averages its keys within non-overlapping blocks of four tokens and scores those block averages against the current query. It selects up to 512 whole blocks, which expand to up to 2,048 individual tokens, and always includes any unfinished final block of up to three tokens. The expensive attention then reads the selected tokens themselves, not the averages. [2] Selection depends on content, not on a fixed window. [1]

It is like skimming a book’s short section summaries before reading chosen passages closely. Good summaries save time, but a summary that hides the one crucial exception can make the search miss it. Sparse attention therefore has to be judged on both its indexing cost and the task quality it keeps.

The attention layers still keep every token in a growing cache, so later queries can choose different blocks. QSA reduces how much of the history the expensive step reads; it does not shrink the history into a fixed-size state. The Gated DeltaNet layers are the ones that keep a fixed-size state. In the figure, doubling the sequence doubles the stored history but leaves the recurrent state unchanged. The figure illustrates only that distinction; it does not model QSA’s selection.

Gates choose what each layer reads

The Gated Residual widens the residual stream, the path that carries each token’s representation up through the layers, into four branches. Each block reads the branches through elementwise gates and writes its result back with a separate weight for each branch. The report drops an extra operation that would mix the branches with each other, to reduce memory traffic. The design is related to, but different from, mHC, described on the GLM-5.3-Flash page, and Attention Residuals, described on the Kimi K3 page. [2]

As a one-number analogy, suppose two branches hold 2 and 8. A gate can read mostly the first branch for one computation and mostly the second for another. With vectors, each coordinate gets its own gate value, which allows finer selection than one weight per branch. In the real model, each branch is normalized, multiplied by gates between zero and one, and the results are averaged. Unlike attention weights, these gates do not have to sum to one. The analogy leaves out the normalization and the learned projections.

The report evaluates the architecture, the optimizer and training stability together instead of relying on training loss alone. That is good practice when reading any release. Lower loss, fewer active weights and a faster kernel are different pieces of evidence, and a deployment still needs measurements of task quality, memory and speed. [2]

Simplifications and limits

• The 51 billion parameters of embedding tables are stored in addition to the 125-billion-parameter backbone.
• The n-gram lookup and the one-number gate examples are teaching analogies, not implementation code.

Follow the idea

Qwen3-NextQwen3.5Kimi K3GLM-5.3-Flash

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Qwen3.8-Flash-Next official model cardSeparate parameter categories, architecture schedule, nominal QSA block budget and custom license; the report additionally specifies incomplete-tail tokens.
2. On the Design of Qwen3.8-Next ArchitectureSections 2.1–2.3 and 3; compressed micro-block indexing, Gated Residual, offloaded n-grams and joint design evaluation.
3. Qwen3.8-Flash-Next weight historyFirst uploaded weight folder August 26, 2026; paper dated August 31.