Llama 4 · Scout and Maverick

What Meta released

Meta released two models on April 5, 2025. Scout has 109 billion parameters in total, of which 17 billion are used for each token. Maverick has 400 billion in total and also uses 17 billion per token. Meta advertises context windows of 10 million tokens for Scout and 1 million for Maverick. Both accept images and text and produce text. [1]

The announcement also previewed Behemoth, a larger model still in training that served as a teacher for the other two. Its weights were not released. A leaderboard result that Meta reported for an experimental chat version likewise does not describe the downloadable models. [2] A result means something only when it is attached to a specific released model.

One shared expert plus one routed expert

A transformer layer’s feed-forward network transforms each token’s vector into a new vector. In a dense model, every token goes through the same feed-forward network. A mixture-of-experts (MoE) layer has many alternative feed-forward networks, called experts, and a small learned router picks which ones each token uses.

Maverick alternates dense layers with MoE layers. Each MoE layer has 128 routed experts and one shared expert. Every token goes through the shared expert and through one routed expert chosen by the router. Llama 4 was the first Llama generation to use experts; the technique itself is older. [2]

The figure cuts the routed experts to eight so every route is visible. Switch between tokens and watch the selected expert change while the shared expert stays on. An expert is a learned transformation, not a specialist in a subject. Routers often group tokens in ways that match no human list of topics.

Research on sparsely gated experts set out the basic bargain: add experts to increase what the model can represent, while running only a few for each token. It also identified the main training risk. If the router sends nearly every token to the same few experts, the others are wasted, and the hardware holding the busy experts becomes a bottleneck. [3]

Why “17B active” does not mean 17B to load

Suppose a layer has eight experts of one million numbers each, plus a shared expert of the same size. A token that uses one routed expert and the shared one touches two million numbers, but the layer stores nine million. Add attention and the dense layers, and the fraction of the whole model that runs for a token changes again.

That is why “17 billion active” does not mean the model loads like a 17-billion-parameter model. All 400 billion of Maverick’s parameters must be held somewhere, often spread across several chips. Different tokens in a batch pick different experts, so data moves between chips and some experts end up busier than others. All of this affects speed, so the fraction shown in the figure is not a speed-up.

Use the total parameter count to estimate weight memory. Treat the active count as a rough guide to computation per token. Then measure real speed with the actual batch size, number format, hardware and context length. [1]

Images become vectors before the model reads them

Llama 4 reads images through a vision encoder based on MetaCLIP. Meta calls its approach early fusion: image and text tokens enter the same model from the start, and the model was pretrained on text, images and video together, instead of having vision added to a finished text model. Early fusion still uses an encoder; raw pixels are not fed in directly. [2]

A vision encoder converts an image into vectors that capture what it shows. MetaCLIP is research on how to collect and balance the image–caption pairs that such encoders are trained on. [4] The resulting vectors make visual information available to the language model. They are not captions, and they do not preserve every pixel.

Take a photo of a red cup beside a ruler, and the question “Is the cup taller than 10 centimeters?” The model has to relate the cup to the ruler’s markings and understand the question. Placing image and text tokens in one sequence lets attention connect them. Reading a tiny tick mark accurately still depends on what the encoder preserved.

Test each part separately

The design raises three separate questions. Does sparse routing produce a good model for the training budget? Does the serving system run the routes efficiently? Does the vision encoder keep the details a task needs? A single leaderboard score cannot answer all three.

The long context windows raise a fourth: how well does the model use information spread across millions of tokens? An advertised limit says what the model accepts, not how well it uses it. Each question can be measured on its own, and that is the most useful way to read any release that combines several new mechanisms.

Simplifications and limits

• Behemoth’s weights were not released.
• The routing figure is scaled down and does not name experts by topic.
• Dividing total by active parameters does not give a speed-up.
• An advertised context length does not mean the model uses every token reliably.

Follow the idea

Llama 3.1 · 8B, 70B, 405BMixtral 8×7B

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Meta’s Llama 4 model cardModel information table: Scout/Maverick totals, active parameters, context, modalities, release and license.
2. The Llama 4 herdApril 5 announcement: routed/shared expert topology, early fusion, encoder and Behemoth release status.
3. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerFoundational sparse gating, conditional computation and load balancing; preceding work, not a Llama 4 invention.
4. Demystifying CLIP DataMetaCLIP data curation and image/text representation learning; does not specify all Llama 4 encoder details.