Mamba

Two ways to carry the past

A transformer generating text stores the keys and values of every earlier token in its KV cache, so each new token can compare itself directly with any of them. The cache grows by a fixed amount with every token. A recurrent model works differently. It carries a state, a fixed-size set of numbers, from one token to the next, and each token updates the state. The state never grows, so it cannot hold everything. It is a summary, and the model has to learn what to put in it.

Mamba belongs to a family called state-space models: recurrent models whose state update is a linear rule with learned coefficients. In earlier models of this family, the coefficients were the same for every token, so the model could not decide to skip a filler word or hold on to a name. Mamba computes the coefficients from the current token. The paper calls this selection, because it lets the model choose what enters the state and what fades. [1]

The figure compares storage for a single layer. The attention line rises with every token; the recurrent state stays flat. The price of the flat line is that recovering an arbitrary earlier token now depends on how well the model compressed it.

Why the update should depend on the input

A memory that holds one number shows the idea. The state h is updated with each input x by the rule h_new = (1 − g) × h_old + g × x. The gate g sets how much the input replaces the old state: g = 0 leaves the state unchanged, and g = 1 replaces it entirely. Feed the memory the sequence 1, 0, 0, 0, 2, 0, 0, 0, where the nonzero numbers are signals worth keeping and the zeros are filler.

With a fixed gate of 0.5, the signal fades. After the first input the state is 0.5, and each zero halves it, to 0.25, 0.125 and 0.0625. A selective gate looks at the input first. It uses g = 1 for a signal and g = 0 for filler, so the state holds 1 until the 2 arrives and then holds 2. Both versions store one number. Only the rule for updating it differs.

Before pressing Step, predict the state after the fourth input under each gate. Real Mamba layers update many numbers at once, and a learned transformation of the input produces their gates; nobody chooses them by hand. The paper’s appendix shows that in the one-number case, Mamba’s update reduces to a gated recurrence like this one. [1]

Training without stepping token by token

Selection created a practical problem. Earlier state-space models, with fixed coefficients, could be trained as one large convolution, an operation GPUs run efficiently. Once the coefficients change with each token, that shortcut no longer applies, and computing the state one token after another would leave most of a GPU idle. [1]

During training, however, the whole sequence is known in advance. Each update has the form “multiply the state by something, then add something”, and two such updates combine into a single update of the same form. For example, “multiply by 0.5, add 1” followed by “multiply by 0.5, add 2” equals “multiply by 0.25, add 2.5”. Updates can therefore be combined in pairs, then pairs of pairs, like a tournament bracket. This is a parallel scan. Mamba runs the scan in a GPU kernel that keeps the expanded state in the chip’s small, fast on-chip memory instead of writing it to slower main memory. [1]

The scan only helps with text that already exists. During generation, the next token is unknown until the model produces it, so the model updates its state one token at a time, at the same small cost for every token. The official code includes the selective state-space layer and the pretrained language models. [2]

What a fixed state cannot do

A fixed-size state cannot hold an unlimited number of unrelated facts exactly. Suppose a prompt lists a thousand random pairs such as “plum → 4417” and later asks which number went with “plum”. Attention can look back at that exact line. A recurrent model must have squeezed all thousand pairs into its state while reading them. Tasks like this, which test exact retrieval and copying, reveal the cost of compression. A good average score on ordinary text does not.

The first release included models from 130 million to 2.8 billion parameters trained on the Pile, a public text collection, plus a separate model trained on SlimPajama. These are base models that predict text; they were not trained to follow instructions. [2] The first weights, for the 130M model, appeared on December 3, 2023. [3] Later recurrent and hybrid models, including Mamba-3 and Qwen3-Next, change the update rule or add attention layers back.

Simplifications and limits

• The one-number memory is a hand-built example. A real Mamba block adds learned projections, a many-number state, discretization, a short convolution and an output gate.
• A fixed recurrent state does not make training memory constant, and it does not give the model unlimited reliable context.
• The storage chart does not predict any speedup.

Follow the idea

Qwen3-NextMamba-3 · MIMO 1.5B

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Mamba: Linear-Time Sequence Modeling with Selective State SpacesSections 3.1–3.5 and Appendix C: selective compression, scan, scalar gate connection.
2. Official Mamba implementation and checkpointsSelective SSM implementation, inference support, and distinct Pile / SlimPajama releases.
3. Mamba-130M release historyModel release commit c4dcd5d on December 3, 2023; official weights.