Mamba-3 · MIMO 1.5B
How a recurrent state can remember more through richer updates
Recurrent models such as Mamba keep a fixed-size state instead of a growing cache, which makes long generation cheap. The open question is how much that state can represent. Mamba-3, published in March 2026 with a 1.5-billion-parameter model released in July, changes three things: how each new input is folded into the state, what kinds of change the state can undergo, and how much useful arithmetic each update performs. The state stays the same size; each update does more with it. [1]
Release: . The paper appeared on March 16, 2026. A commit titled “Mamba-3 public release” (bc6b5d0) added the 1.5B MIMO weights to the official repository on July 28, 2026. The repository is public, although one sentence in its README still describes it as private.Model: 1.50B · attention-free · MIMO rank 4Weight license: Apache 2.0
The paper and the downloadable model
The Mamba-3 paper appeared on March 16, 2026. The page is dated July 28, when a commit titled “Mamba-3 public release” added the weights of a 1.5-billion-parameter model to the official repository. The paper date marks the method; the July date marks the model you can download. [1] [2]
The model card describes a pretrained language model with no attention layers: 24 layers, a MIMO rank of four (explained below) and the Apache 2.0 license. It is a research checkpoint, not a chat assistant. It was trained on 100 billion tokens of FineWeb-Edu, a collection of educational web text, with a 2,048-token context. A fixed-size state can in principle keep running indefinitely, but that does not make the model reliable on arbitrarily long inputs, and the paper’s benchmark results do not automatically apply to this download. [3]
The basic mechanism is a recurrence: compute the next state from the old state and the incoming token. A fixed-size state can keep evolving forever, but every update has to decide what survives.
Using both ends of each step
Mamba-3 changes how each input enters the state. Its exponential-trapezoidal update decays the old state and adds contributions from both the previous token’s input and the current one, where the original Mamba used only the current one. In simplified form: new state = a × old state + b × previous input + c × current input. The real coefficients are learned and more richly structured than this. [1]
Take an old state of 10, a previous input of 2, a current input of 6 and a decay of 0.8. Weighting the two inputs equally, with the previous one also decayed, gives 0.8 × 10 + 0.4 × 2 + 0.5 × 6 = 8 + 0.8 + 3 = 11.8. Using only the current input at full weight gives 0.8 × 10 + 6 = 14. The point is that the update has a new way to blend neighboring inputs, not that either number is better.
The name comes from numerical methods. State-space models are defined in continuous time and must be converted to discrete steps, one per token. The trapezoidal rule approximates a quantity over an interval using its values at both ends, which is why both inputs appear. The continuous-time view supplies a principled family of update rules, which training then tunes.
One update, more internal operations
Rotating the state as well as shrinking it
In the original Mamba, an update can only shrink each stored value by a decay factor. Mamba-3 adds rotation. It treats pairs of numbers in the state as the two coordinates of a point and rotates that point by an angle that depends on the input; mathematically, this is equivalent to a state made of complex numbers. The paper relates this to rotary position embeddings, with one difference: the angle depends on each token’s content, not only on its position. [1]
Take a two-number state [1, 0]. Rotating by 90 degrees gives [0, 1]; again gives [−1, 0]; again gives [0, −1]. The length stays 1 while the direction cycles through four distinct values, so the state can track which of four stages it is in. Multiplying by a positive decay factor alone can never produce that cycle.
The cycle is a made-up geometric example; no trained channel is claimed to store a four-step counter. It shows that rotation adds kinds of change the state could not make before. Because the angle depends on the input, different events can turn the stored direction by different amounts.
Doing more arithmetic while the state is loaded
MIMO stands for multiple-input, multiple-output. The authors’ technical post explains the motivation from hardware. During generation, each recurrent step spends much of its time moving the state in and out of memory rather than computing with it, so extra arithmetic can fit into that waiting time at little cost. How much fits depends on the implementation; it is not free in general. [4]
Suppose reading and writing the state takes 10 time units, and the arithmetic takes 2 units that overlap with the data movement. Raising the arithmetic to 4 might leave the total near 10. Raising it to 20 would not.
MIMO uses that slack by updating and reading the state along several projected directions at once, instead of one. Think of taking several views of the same stored array while it is in hand. It does not generate several tokens at once, and it does not multiply the state’s size by the MIMO rank. [4]
A better state is still a finite state
The figure compares a made-up recurrent state of 256 numbers with a cache that adds 64 numbers per token. Doubling the sequence length doubles the cache and leaves the state unchanged. A real Mamba-3 also needs memory for weights, projections and temporary buffers, so this is not its measured footprint.
The authors note that retrieving specific earlier details remains hard for purely recurrent models, and they study hybrids that add attention back. Richer updates let a fixed-size state encode more; they do not turn it into a complete record of everything it has read. [4]
Mamba-3 therefore works within one constraint, a fixed-size state, and improves three things inside it: how neighboring inputs are combined, what kinds of change the state can undergo, and how well each update uses the hardware.
The state’s size does not depend on sequence length
Simplifications and limits
- The page is dated by the official public-release commit of the weights, not by the March paper.
- The download is a base research checkpoint; the paper’s benchmark results are not automatically results for it.
- The arithmetic examples and state sizes do not reproduce the trained model, and they do not show unlimited retrieval.
- Whether extra arithmetic fits within the same generation time depends on the workload.
Sources and further reading
Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.
- Mamba-3: Improved Sequence Modeling using State Space PrinciplesMarch 16 paper; Sections 3.1–3.3: exponential-trapezoidal update, complex rotations and MIMO.
- Official Mamba-3 MIMO release commitCommit title Mamba-3 public release, dated July 28, 2026; public API independently checked private=false and gated=false.
- Official Mamba-3 MIMO 1.5B model cardCheckpoint architecture, 24 layers, MIMO rank 4, pretraining configuration, public weights and Apache 2.0.
- Mamba-3 technical post by its authorsArchitecture, MIMO arithmetic intensity, inference measurements and retrieval limitations; posted March 17.