Falcon-H1 · 0.5B to 34B

Six sizes of one hybrid design

TII announced Falcon-H1 on May 21, 2025, with base and instruction-tuned versions at 0.5B, 1.5B, 1.5B-Deep, 3B, 7B and 34B parameters. The detailed technical paper followed in July. [1] [2]

Here, hybrid means a model that combines transformer attention with a state-space model (SSM), in this case Mamba-2, a successor to the original Mamba design. Both are learned parts of one language model; the model does not switch between two assistants.

Think of reading a meeting transcript. Keeping track of the current topic suits a compact summary that updates as you read. Finding the exact date someone mentioned an hour ago suits looking back at the precise earlier words. That is the intuition for having both. Nothing guarantees that the trained model divides the work this way.

Both branches read the same input

In Falcon-H1, each block’s input goes to an attention branch and an SSM branch at the same time. The two outputs are joined end to end and passed through a learned projection that mixes them. In the more common alternating design, one layer’s output becomes the next layer’s input, so attention and recurrence take turns. Parallel hybrids had been tried before; Falcon-H1 adopted the design and studied it at several scales. [3]

In symbols: mix(x) = W · [attention(x); SSM(x)]. The semicolon means concatenation, placing the two vectors end to end, and W is a learned matrix. The formula leaves out normalization, residual connections and the rest of the block.

Suppose attention outputs [2, 1] and the SSM outputs [5, −1]. Concatenated, they become [2, 1, 5, −1]. A projection row of [1, 0, 0.5, 0] produces 1 × 2 + 0.5 × 5 = 4.5. The projection can draw on both branches without requiring their outputs to mean the same thing coordinate by coordinate.

A hybrid keeps both kinds of memory

The SSM branch updates a fixed-size state. The attention branch keeps a cache that grows with every token. A model with both is therefore not constant-memory: its memory still grows with the input. The design question is how much width and computation to give each branch.

In the figure, the recurrent state holds 256 numbers and the attention cache adds 64 numbers per token. At 128 tokens, the two together hold 256 + 128 × 64 = 8,448 numbers; at 256 tokens, 16,640. The figure draws the two parts separately, and its sizes are made up rather than taken from Falcon-H1.

The report treats the widths of the attention and SSM branches as settings to tune. Giving attention a smaller share reduces its cost while keeping a direct route back to earlier tokens. Whether a smaller attention branch is still good enough has to be measured; the diagram cannot predict it. [3]

The same size can be built different ways

The team’s technical post explains the 1.5B-Deep variant. It has about as many parameters as the regular 1.5B model, spread across many more, narrower layers. In the team’s measurements the deeper model scored higher but ran less efficiently. The post also describes tuning the ratio of attention to SSM heads at each size. The result is a family of configurations, not one ratio that suits every size. [2]

A fixed amount of computation can go into a few wide layers or many narrow ones. Many narrow layers apply more transformations in sequence; a few wide ones do more work in parallel at each step. The parameter count alone does not tell you how many steps run in sequence, how much data moves through memory, or how long a response takes.

“Parallel” in an architecture diagram does not guarantee that the two branches run at the same moment on the hardware. Whether they overlap depends on the software kernels and how the work is scheduled. The design makes overlap possible; an implementation has to deliver it.

Choosing a checkpoint and measuring it

The model card for the 7B base model describes a decoder-only hybrid released under TII’s own Falcon-LLM license. A base model continues text; the instruction-tuned version is trained to respond to requests. When judging the architecture, compare like with like, and do not credit the architecture with behavior that comes from instruction tuning. [4]

For a useful comparison, measure separately the time to answer a short question, the time to read a long prompt, the speed of long generation, and accuracy at retrieving details from long inputs. A model can improve on one of these without improving on the others. Report the context length and batch size beside any memory or speed figure.

Falcon-H1’s main idea is where attention and recurrence meet: inside each block, side by side. That opens a continuous range of designs, from mostly attention to mostly recurrence, and each point in that range still has to be measured as a whole system.

Simplifications and limits

• A hybrid’s memory includes both a fixed-size state and an attention cache that grows with the input.
• The block diagram shows the architecture; it does not mean the two branches run simultaneously on the hardware.
• The memory figure uses made-up sizes and predicts neither real memory use nor quality.
• Size labels such as 7B are rounded names, not exact parameter counts.

Follow the idea

MambaPhi-4-mini-flash-reasoningQwen3-Next

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. TII launches Falcon Arabic and Falcon-H1May 21, 2025 public launch announcement; performance marketing is not reproduced.
2. Falcon-H1 technical release postModel links, family variants, parallel heads, depth tradeoff and training-design discussion.
3. Falcon-H1: A Family of Hybrid-Head Language ModelsArchitecture section and Figure 1: same-input parallel branches, concatenation/projection, adjustable widths and configuration table.
4. Official Falcon-H1-7B-Base model cardReleased base checkpoint, causal hybrid architecture and Falcon-LLM license.