Mixtral 8×7B

What an “expert” is

Each transformer layer does two jobs. Attention moves information between token positions. A feed-forward network then transforms each token’s representation on its own; in most models, this part holds the majority of the weights. Mixtral keeps a single attention block per layer, shared by all tokens, and replaces the single feed-forward network with eight of them. This design is a mixture of experts (MoE), and each of the eight networks is called an expert. [2]

Unlike Mistral 7B, Mixtral uses full attention over its whole 32,768-token context, with no sliding window. [2]

The name “8×7B” suggests eight complete 7-billion-parameter models, which would total 56 billion. That is not the design. Only the feed-forward networks come in eights; attention and the other shared parts exist once. That is why the total is 46.7 billion.

How a token picks two experts

At every layer, a small router scores the eight experts for the current token and keeps the two highest. Both chosen experts process the token, and their outputs are added together, weighted by the router’s scores. The next token may pick a different pair, and the same token may pick a different pair at the next layer. [2]

Here is the combining step with two-number vectors. Expert A outputs [2, 0], expert B outputs [0, 3], and the router gives them weights 0.75 and 0.25. The layer’s output is 0.75 × [2, 0] + 0.25 × [0, 3] = [1.5, 0.75]. The router blends internal vectors, partway through a calculation; the experts are not voting between finished answers.

In the figure, switch between Token A, Token B and Token C to see different experts selected, then change how many experts each token uses. Using more experts per token adds computation; the number of experts stored stays at eight. The routes are made up for illustration; they are not Mixtral’s measured choices.

All experts must stay in memory

Any token might pick any expert, so a server has to keep all eight experts of every layer ready. Routing reduces the arithmetic for each token. It does not reduce the memory needed for the weights. Moving idle experts out to slower memory saves space but adds time spent fetching them back.

The bars compare the two parameter counts on the same scale. Their ratio is not a speed-up. Attention, routing, moving data between chips, batching and hardware efficiency all add time that the parameter count leaves out. A common mistake is to read “12.9 billion active” as “runs like a 12.9-billion-parameter model”. It computes like one per token, roughly, but it must be stored like a 46.7-billion-parameter model. [1]

Storing 46.7 billion numbers at two bytes each takes 93.4 GB, for the weights alone. The KV cache, temporary values and software overhead come on top. Storing the weights in fewer bits reduces this figure, but a smaller format needs its own testing for accuracy.

Experts do not specialize by subject

The word “expert” suggests one network for biology and another for law. The Mixtral paper looked for that pattern and did not find it. When the authors tracked which experts handled text from different subjects, the choices were spread similarly across subjects. What they did find was structure in the text itself: consecutive tokens often went to the same experts, and some choices tracked syntax, such as indentation in code. [2]

That makes sense once you remember what an expert does. It transforms a token’s internal representation, and a biology sentence and a legal sentence may need the same transformation, for punctuation for instance. This page numbers the experts instead of naming them, because the evidence supports no names.

Mixtral did not invent the mixture of experts. Earlier research on sparsely gated MoE layers developed the routing idea and the problem of spreading tokens evenly across experts. Mixtral showed that the approach could produce a strong language model with downloadable weights. [3]

The larger Mixtral uses the same design

Mistral AI released Mixtral 8×22B in April 2024. It has 141 billion parameters in total and 39 billion active per token, with a longer context window. It still routes each token to two of eight experts per layer, so it scales up the same design. [4]

The two sizes show two separate settings in an MoE design. Adding more or larger experts increases what the model can store and represent. Sending each token to more experts increases the computation per token.

The model card lists the base model and the instruction-tuned model as separate downloads. The routing design describes how computation is organized; how the model behaves in a conversation depends mostly on the training that follows pretraining. [5] The DeepSeek pages that follow change the design further, adding experts that every token uses alongside many smaller routed ones.

Simplifications and limits

• Experts are numbered, not named; the paper found no evidence of subject specialization.
• The ratio of active to total parameters is not an estimate of speed.
• The release date is the torrent posting; the official announcement is dated December 11.

Follow the idea

Mistral 7BDeepSeek-V2DeepSeek-V3

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Mistral: Mixtral of expertsFormal December 11 announcement, public release and parameter counts.
2. Mixtral of Experts technical reportSections 2 and 5: top-two routing, shared attention and observed routing patterns.
3. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerPrior conditional-computation mechanism and load-balancing problem.
4. Mistral: Cheaper, Better, Faster, StrongerMixtral 8×22B companion specifications and release.
5. Official Mixtral 8×7B model cardWeights, license and base checkpoint identity.
6. Mistral AI original torrent announcementOriginal public-weight torrent announcement, December 8 at 15:44:17 UTC; author, content and timestamp verified through X’s official syndication JSON.