Gemma 3 · 1B, 4B, 12B, 27B
How mostly local attention made room for images and long inputs
Every token a model reads adds to the cache in every attention layer, so long documents and images, which become many tokens, are expensive to hold in memory. Gemma 3, released by Google in March 2025, made most of its layers look only at the most recent 1,024 tokens and let just one layer in six see the whole input. That kept the cache manageable while the larger models gained 128,000-token inputs and the ability to read images. This page explains that layer pattern and how images get into the model.
Release: . Google released the Gemma 3 weights on March 12, 2025. The 1B model differs from the larger sizes, as described below.Model: 1B text-only · 4B, 12B, 27B image/textWeight license: Gemma Terms of Use; custom terms and use restrictions
Where a running model’s memory goes
Google released Gemma 3 on March 12, 2025, in four sizes. The 1B model reads text only and accepts up to 32,000 tokens. The 4B, 12B and 27B models accept images as well as text, and up to 128,000 tokens. [1]
The memory a running model needs falls into separate parts: the weights, the cache for each conversation, temporary values computed along the way, and software overhead. Shrinking one part leaves the others unchanged. A model with modest weights can still need a large cache when many people send long inputs.
The cache exists because each attention layer stores keys and values for earlier tokens so that new tokens can consult them. If every layer keeps the entire history, each new token enlarges every layer’s cache. Gemma 3’s design starts from a question: does every layer need direct access to the whole history?
Most layers look only nearby
Gemma 3’s answer is no. Its layers repeat a pattern of five local layers, each seeing only the previous 1,024 tokens, followed by one global layer that sees everything. Gemma 2 had alternated one local and one global layer, with a 4,096-token window. Only the global layers need to store the whole history. [2]
The arithmetic shows the effect. Take a six-layer model reading 16,384 tokens. If all six layers are global, they store 6 × 16,384 = 98,304 token positions. With five local layers capped at 1,024 positions and one global layer, they store 5 × 1,024 + 16,384 = 21,504, under a quarter as many. The calculation ignores head sizes and other details. It shows why the cache now grows much more slowly as the input gets longer.
The figure shows one layer at a time. Move the query position to the right. In Local mode, older positions drop out of view; in Full mode, they stay visible. Gemma 3’s stack is mostly local layers with an occasional full one.
Which earlier positions can this layer see?
How distant information still arrives
Suppose the first paragraph of a long document defines “the blue plan” and the last paragraph asks which plan fits the budget. A global layer can pull the definition into the representation of a current token. Local layers above it then work with that representation; they do not need to look back at the first paragraph themselves. That is the route by which distant information can reach the answer. Whether the model actually uses the route has to be tested.
Accepting a long input is different from using it well. A 128,000-token limit gives the model the chance to use a distant passage. It does not guarantee that the model notices the passage, reconciles it with contradicting text, or retrieves equally well from every position.
The report tested the design directly. It varied the ratio of local to global layers and the window size, then measured quality and memory, and found that the local-heavy pattern cost little quality while greatly reducing cache memory. [2] Experiments that change one setting at a time are better evidence than crediting every improvement to a single feature.
How images become tokens
The 4B, 12B and 27B models read images through a vision encoder based on SigLIP, a model trained to match images with their text descriptions. [3] The encoder converts an image into vectors that describe its visual content. It does not read text in images perfectly, and it does no reasoning of its own.
Gemma 3 compresses each encoded image into 256 soft tokens. These vectors enter the language model in the same way as word tokens, but they come from the encoder, not from the vocabulary. The encoder works on a fixed 896 × 896-pixel square, so a large or oddly shaped image loses detail when shrunk to fit. An optional method called pan-and-scan cuts such an image into several crops, encodes each one and passes them all to the model. More crops preserve more detail but add more tokens. [2]
Picture a photo of a wide restaurant menu. Shrunk into a small square, the section headings may stay readable while the prices blur. Crops give the prices more pixels. The language model still has to match each price with the right dish, so seeing and reasoning are two separate places where an answer can go wrong.
Keep the detail before asking the question
Fewer bits for the weights
Google also released quantized versions, which store each weight in fewer bits. [4] Quantization shrinks the weights. It does not shrink the cache for long inputs, and it can lower quality, so each quantized model needs its own testing.
The storage figure takes 12 billion weights. At 16 bits each, they need 24 GB; at 4 bits, 6 GB. That is arithmetic on the weights alone. Scale factors, parts kept at higher precision, the vision encoder and working memory make a real model and runtime larger.
The two interactive figures act on different parts of the memory budget. The attention figure changes how much history each layer keeps. The storage figure changes how many bits each weight uses. The savings combine, and neither replaces the other.
Weight precision is a separate setting
Simplifications and limits
- The 1B model reads text only and has a shorter context than the larger models.
- Neither figure measures Gemma 3’s speed or whether it fits on a particular device.
- A context limit does not guarantee reliable retrieval from every position.
Sources and further reading
Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.
- Google’s Gemma 3 model cardModel variants, modalities, context limits, weights and license.
- Gemma 3 Technical ReportSections 2, 2.1 and 5.2–5.3: 5:1 attention, 1,024-token windows, soft visual tokens and crop handling.
- Sigmoid Loss for Language Image Pre-TrainingSections 2–3: pairwise image/text objective and image representation learning.
- Introducing Gemma 3March 12 release, supported family and quantized deployment options.