Gemma 4 · E2B, E4B, 26B-A4B, 31B, later 12B

One family, several architectures

Google released the first Gemma 4 weights on April 2, 2026. The initial family had four members: E2B, E4B, a 26B mixture-of-experts model and a 31B dense model. The 12B model without separate encoders was added on June 3; it is a later design and does not describe the April models. [1] [2]

The technical report names the mixture-of-experts model 26B-A4B: about 26 billion parameters in total, with about 3.8 billion active per token. E2B and E4B use per-layer embeddings, a technique from Gemma 3n in which large embedding tables sit outside the main computation and can be stored separately. Their names count only the “effective” parameters, about 2.3 and 4.5 billion, while their totals are roughly 5 and 8 billion. Effective, active and total are three different ways of counting. [3]

Imagine two inventories of a model. The first lists every stored number. The second lists only the numbers used to process one token. Sparse routing shrinks the second without changing the first, and embeddings stored apart from the main computation need a third description. Check which count a name uses before drawing conclusions about memory or speed.

The 12B model reads images and audio directly

In the usual design, a separate image network, the encoder, transforms pixels into vectors before the language model sees them. Gemma 4 12B drops that network. Small learned projections map image patches and short chunks of raw audio straight into the language model’s embedding space. Google’s June announcement calls this a unified, encoder-free architecture. [2]

Encoder-free does not mean the input arrives unprocessed. It still has to be cut into pieces of a fixed shape. The report describes image patches of 48 × 48 pixels, each with red, green and blue values, and audio chunks of 40 milliseconds. A projection converts each piece into a vector of the right size, and position information records where each image patch came from. [3]

Take one 48 × 48 patch. Its color values form a list of 48 × 48 × 3 = 6,912 numbers. Multiplying that list by a learned matrix gives one token vector, and the same happens for every patch. At this point a patch is just numbers; recognizing what it shows is left to the transformer.

Moving work does not remove it

A projection is far smaller than a deep vision encoder, but the transformer must now learn to build useful representations from raw pieces. Saving one component can increase the demands on another, in training or at run time. “Unified” says where the work happens; it does not prove that every task becomes cheaper.

Consider reading a handwritten total on a receipt. The model must separate ink from paper, recognize the digits, connect the total to its label and answer the question. An encoder-based system does part of that work before the language model starts. An encoder-free system learns more of it inside the language model.

This is a comparison of architectures. The diagram contains no measured accuracy or speed, so it cannot say which design is better for a particular camera feed, document collection or device.

Training with the rounding in view

On June 5, Google released Gemma 4 checkpoints prepared with quantization-aware training (QAT), in formats including Q4_0 and others aimed at phones. QAT simulates the rounding of low-precision storage during training, so the weights can adapt to the distortion they will face after deployment. The method is established; Gemma 4 applies it. [4]

Suppose a weight of 0.37 must be stored on a coarse grid that allows only 0.4. Rounding only after training surprises the model. Showing it the rounding during training lets the other weights adjust to compensate. QAT cannot preserve every network perfectly, but it moves training in a useful direction.

The figure shows ideal storage for 12 billion weights: 24 GB at 16 bits, 12 GB at 8 bits and 6 GB at 4 bits. Real Gemma files add format overhead and keep some parts at higher precision. The figure makes no claim that any particular device can run the full model.

A small drafter proposes several tokens

The July technical report also describes a drafter for speculative decoding: a small network that predicts several upcoming tokens using the main model’s internal representations and cached attention data. The main model then checks the proposals. A draft is only a suggestion, and the main model has the final say. [3]

Suppose the drafter proposes “the answer is six”. The main model checks all four tokens in one pass and accepts “the answer is” but not “six”. The three accepted tokens are kept, and the main model supplies its own next token in place of “six”. When many drafts are accepted, one expensive pass of the main model yields several tokens.

The gain depends on how often drafts are accepted, how much the drafter costs, and the hardware. Gemma 4 thus shows several separate routes to efficiency: running only some weights per token, a simpler path for images and audio, fewer bits per weight, and cheaper proposals during generation. A fair comparison says which route changed and measures its effect on the same workload.

Simplifications and limits

• The page is dated by the April family launch; the 12B model and the QAT checkpoints are dated June.
• Effective, active and total parameter counts are not interchangeable.
• The storage figure is an ideal calculation and makes no claim about fitting on a device.
• Only the 12B model is encoder-free; the other Gemma 4 models keep their encoders.

Follow the idea

Gemma 3 · 1B, 4B, 12B, 27BPhi-4-reasoning-vision-15B

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Introducing Gemma 4April 2, 2026 public-weight family and Apache 2.0 license.
2. Introducing Gemma 4 12BJune 3, 2026 encoder-free addition and downloadable weights.
3. Gemma 4 Technical ReportSections 2, 2.3 and 2.6: parameter accounting, direct projections and MTP drafter; report dated July 2.
4. Gemma 4 with quantization-aware trainingJune 5 release of QAT weights and explanation of simulated quantization.