Phi-4-mini-flash-reasoning

How a hybrid decoder reuses memory instead of repeatedly reading the whole past

A reasoning model may write tens of thousands of tokens for one answer, and each new token has to look back over everything before it. That makes long answers slow. Phi-4-mini-flash-reasoning, a 3.8-billion-parameter model released by Microsoft in July 2025, uses a new hybrid architecture called SambaY to cut the repeated work. Most of its layers use a cheap recurrent summary or a short attention window. The full history is stored once and shared, and some layers reuse an earlier layer’s result instead of reading the history again. The model card reports up to ten times the throughput of its predecessor, Phi-4-mini-reasoning, on long generation. [4]

5 sections · about 8 minutes · Memory

Release: 2025-07-09. Microsoft released the weights and announced the SambaY paper on July 9, 2025.Model: 3.8B hybrid · 64K contextWeight license: MIT

Why long answers are slow

Microsoft released Phi-4-mini-flash-reasoning on July 9, 2025, with a 64,000-token context. The announcement describes its SambaY architecture, which mixes state-space layers, sliding-window attention and memory shared between layers. “Flash” is part of the model’s name; it does not mean every operation takes constant time. [1]

Imagine the model writing a long proof. At first the history is short. Thousands of tokens later, each new token must read through a much longer history, which means moving a lot of data out of memory. If many layers each read the full history, the model repeats that expensive work many times for every token.

Generation has two phases. In prefill, the model reads the prompt, many tokens at once. In decode, it writes the answer one token at a time. A design that speeds up reading the prompt does not automatically speed up writing a long answer. This model was designed for long answers, so decode is the phase that matters.

A running summary of fixed size

A recurrent layer keeps a state, a fixed-size vector that it updates as each token arrives. In Mamba, the update depends on the current token, so the layer can choose what to keep and what to let fade. SambaY uses Mamba layers from this earlier work. [2]

Here is an analogy. While reading a long stream of numbers, keep only a running total and a count. You hold two numbers however long the stream is, and you can compute the average at any time. But you cannot recover the seventh number. A summary saves memory by giving up direct access to the details.

The figure compares a made-up state of 256 numbers with an attention cache that adds 64 numbers per token. At 128 tokens the cache holds 8,192 numbers. Move the slider: the state stays flat while the cache grows. These sizes are for illustration; they are not Phi’s.

A fixed summary beside a growing record

Made-up sizes for two separate memory components. Phi-4-mini-flash-reasoning is a hybrid that still has an attention cache, so this is not its total memory. A fixed-size state caps one part of memory by compressing the past.

Keeping a summary and a full record

The SambaY paper splits the model into two halves. The first half, the self-decoder, alternates Mamba layers with sliding-window attention and ends with one full-attention layer. That layer’s keys and values are stored once, as a cache shared with the second half. The second half, the cross-decoder, has attention layers that read the shared cache instead of building their own, alternating with gated memory units (GMUs) that do not read the history at all. GMUs replace half of the cross-decoder’s attention layers. The design builds on YOCO (“you only cache once”), an earlier Microsoft architecture. [3]

The two kinds of memory are good at different things. To recall a specific identifier from far back, it helps to look at the stored tokens directly. To keep track of what the recent text is about, a running summary may be enough, with no need to revisit every token.

The flat line in the previous figure is therefore not the whole model’s memory. The model keeps one full attention cache. What it saves is the repeated reading of that history in layer after layer.

Reuse results across the decoder

Simplified SambaY layout. The real model interleaves the layers and includes extra projections, feed-forward layers and a variant of attention called differential attention. Some later layers read the shared cache; others reuse a result computed earlier.

Reusing an earlier result through a gate

A gated memory unit takes two inputs: the current layer’s representation, and a stored representation from an earlier layer, which in SambaY is the output of the last Mamba layer. Learned projections of the current representation produce a gate: one value per channel that controls how much of the stored representation passes through. Multiplying the two channel by channel gives the unit’s output. [3]

Take a stored vector [2, 4, 6] and a gate [1, 0.5, 0]. Multiplying channel by channel gives [2, 2, 0]: the first channel passes in full, the second at half strength and the third is blocked. The published unit adds learned projections and a smooth activation function called SiLU, but the idea is the same. The layer reuses information already computed, at a small fixed cost, without searching the history again.

The channels are learned numerical features, not labeled memories. Calling one a “math memory” would need evidence from the trained model, not just a convenient reading of the diagram.

Speed claims need their conditions

The model card’s throughput gain comes from a specific test. [4] The paper’s timing experiment used randomly initialized weights, a single A100-80GB GPU and a customized version 0.7.3 of the vLLM serving software, with prompts of about 2,000 tokens and generated answers of about 32,000. Random weights make it easy to control how long the answers are, but they mean the test measured the architecture and software under load, not useful answers per second from the trained model. [3] The released weights use the MIT license. [4]

A clean comparison keeps the task, the prompt and answer lengths, the hardware and the software the same, then measures quality and speed together. A faster model that loses information the task needs is no improvement, and better accuracy does not help if the answer arrives too late.

The main idea is structural. Not every layer needs to read the full history: layers can share one cache, and some can reuse an earlier layer’s output through a learned gate. The remaining attention keeps abilities that a fixed-size summary alone may lack.

Simplifications and limits

The flat line in the memory figure describes one component only; the real model still stores a full attention cache.
The gate example shows the arithmetic idea, not the full published formula.
The reported throughput depends on the workload and software; it is not a general speed-up.

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

Introducing Phi-4-mini-flash-reasoningJuly 9 release, 3.8B size, 64K context and hybrid model positioning.
Mamba: Linear-Time Sequence Modeling with Selective State SpacesSection 3: input-dependent recurrent state selection; not a Phi-specific invention.
Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long GenerationSection 2 and Figure 1: SambaY, shared KV, GMU equations and reuse of SSM readouts.
Microsoft’s Phi-4-mini-flash-reasoning model cardReleased weights, MIT license and conditional 2K-input/32K-output throughput evaluation.