Register tokens in Vision Transformers

How an image becomes a sequence of tokens

A Vision Transformer (ViT) cuts an image into a grid of square patches, usually 14 or 16 pixels on a side. A 224-pixel image cut into 16-pixel patches gives a 14 × 14 grid of 196 patches.DINOv2 uses 14-pixel patches, which gives a 16 × 16 grid of 256 patches at 224 pixels. For legibility, the simulation on this page uses the 14 × 14 grid everywhere, including where it is labeled DINOv2. Each patch is flattened and multiplied by a learned matrix, which turns it into a vector of d numbers (768 for ViT-B, 1,024 for ViT-L). A positional embedding is added so the model knows where the patch came from. These vectors are the patch tokens.

One more learned vector, the class token [CLS], goes in front of the patches. It has no pixels of its own. Through attention it gathers a summary of the whole image, and its final state is the image representation used for classification. The sequence is therefore 197 tokens long: [CLS] at position 0, then the patches in reading order, left to right and top to bottom, at positions 1 to 196.

The sequence then passes through a stack of transformer blocks, 12 in ViT-S and ViT-B and 24 in ViT-L. In each block, self-attention lets every token read from every other token and write the result into its own vector. Nothing in the architecture requires the token at position 92 to keep describing patch 92. If the network finds a better use for that slot, training is free to put it there.

A few patch tokens with ten times the norm

The norm of a token is the length of its vector: the square root of the sum of its squared entries. Darcet and colleagues found that in DINOv2 a few patch tokens leave the network with norms about ten times larger than the rest, and that these tokens are the bright specks in the attention maps.Paper, §1 and Fig. 3. In DINOv2 ViT-g most patch norms lie between 0 and 100, and 2.37% exceed 150, a cutoff the authors chose by hand and note varies between models. The paper calls them high-norm or outlier tokens. This page calls them artifact tokens.

They tend to occur in uniform background, such as sky, walls or roads, where a patch resembles its neighbors.Fig. 5a: measured right after patch embedding, patches that become high-norm tokens are very similar to their four neighbors. Appendix A: they are also more common near the image border. The authors interpret this redundancy as an opportunity to reuse the token while losing little information about the image.

When and in which models they appear

The paper reports three conditions. The outliers separate from the other tokens partway through the network, around layer 15 of the 40-layer ViT-g. They appear only after about a third of training. And of six DINOv2 sizes, from ViT-Tiny to ViT-giant, only the three largest (ViT-L, ViT-H and ViT-g) have them.Paper, §2.1 and Fig. 4. Training length is not simulated on this page.

Size is not the whole story. The paper also finds artifacts at ViT-B in models trained with the supervised DeiT-III recipe and with OpenCLIP, and none in the original DINO. The authors say plainly that they could not determine which aspects of training cause them.§2.2: “we have not been able to fully determine which aspects of the training led to the appearance of artifacts in different models.” OpenCLIP and DeiT-III show outliers at both ViT-B and ViT-L (Fig. 2).

What the artifact tokens hold

To find out what information a token still carries, the authors trained linear probes: small linear models fitted on frozen token vectors. If a probe predicts a patch’s position, that information is accessible through a linear readout. A weaker probe score does not prove that all information about the patch is gone. They probed normal and artifact tokens of DINOv2 ViT-g for three things: where the patch was, what pixels it held, and what the whole image showed.

Artifact tokens do worse on both local tasks and better on both global ones. The gap is widest on fine-grained aircraft classification: a probe on one normal patch token gets 17.1% right, a probe on one artifact token 79.1%, close to the 87.3% of [CLS], the token built for the job.

The authors’ hypothesis is that a large, well-trained model learns to recognize patches whose content is redundant, discards their local information, and uses their token slots to store, process and retrieve global information. They do not consider this computation harmful in itself. The problem is where it happens: inside patch tokens, which downstream methods read as a map of the image.

That explains which tasks suffer. Classification reads [CLS] and barely notices. Segmentation, depth estimation and object discovery read the patch tokens directly, and an overwritten token no longer describes its patch. The authors expected this to cost accuracy on dense prediction, and the results below bear that out most clearly for object discovery. The investigation started with such a mismatch: the object-discovery method LOST works well on the original DINO and poorly on DINOv2.§1: “we observed that DINOv2 is surprisingly incompatible with LOST.” LOST (Siméoni et al., 2021) finds the main object from the similarity between patch features.

Registers: extra tokens for global computation

A register starts as a learned vector added to the input sequence, with no image patch of its own. Its initial vector is shared across images; its state becomes image-dependent as transformer blocks update it. Registers pass through every transformer block like any other token, and their outputs are thrown away at the end: only [CLS] and the patch tokens are used, in training and at inference. The mechanism comes from Memory Transformers, where it improved machine translation.Burtsev et al., 2020, cited in §2.2 of the paper.

In the official DINOv2 code the registers sit after [CLS] and before the patch tokens, and they are inserted after the positional embeddings are added, so they carry no position.prepare_tokens_with_masks in dinov2/models/vision_transformer.py. The paper’s text says the registers are “appended”. Without positional embeddings, where a register sits in the sequence makes no difference to attention. Their order in the sequence matters only to downstream code, which drops the first 1 + n outputs and reshapes the rest into the patch grid.

Discarding a register’s final output does not erase its earlier influence. In one block it can read from patch tokens; in a later block, patches and [CLS] can read from its updated state. Training can therefore adjust the initial register vector through its effect on the retained outputs.

In the paper’s trained models, the high-norm outputs occur among the registers instead of the patch tokens. These are separately trained models: the comparison does not track a fixed quantity moving between slots.

The paper finds the same transfer in what the tokens hold. In a model with one register, a probe on the register classifies aircraft with 71.1% accuracy, close to the 73.3% of an artifact token in the same model trained without registers. The ordinary patch tokens keep the same local information with or without registers.Appendix D, Tables 4 and 5. Position top-1 on normal patches is 66.3% without registers and 65.8% with four; reconstruction L2 error 15.9 and 16.0.

In the tested DINOv2 ViT-L models, one register removes the visible artifacts. Adding more keeps improving ImageNet accuracy, while segmentation and depth have an optimum; the paper uses four for all its main results.Fig. 8: DINOv2 ViT-L/14 trained with 0, 1, 2, 4, 8 and 16 registers. The paper reports less than 2% extra model FLOPs with four registers in its configurations. This is not a universal overhead: it depends on the number of patches and the model width.Appendix B and Fig. 12. For this page’s 197-token illustration, four registers add 4 × 1,024 = 4,096 learned parameters to ViT-L. Token-wise work grows by 201/197 − 1 ≈ 2.0%; the attention score matrix grows by (201/197)² − 1 ≈ 4.1%. Those are component costs, not measured latency.

The registers do not all behave alike. In the paper’s attention maps, individual registers attend to different objects or regions of the image, although nothing in training asked them to.Fig. 9 and Appendix D.3. One register of a DINOv2 model attends mostly to the image border on average.

What registers change downstream

The authors trained three model families with and without four registers: DeiT-III, supervised on ImageNet-22k labels; OpenCLIP, trained on image–text pairs; and DINOv2, self-supervised. The high-norm patch artifacts disappear in all three. The first three tasks use linear evaluation on frozen features. LOST is a separate object-discovery procedure built from patch-feature similarities; it does not train a linear prediction head.

ImageNet accuracy moves by at most 0.5 points. ADE20k segmentation and NYUd depth improve in most cases, with small regressions for DeiT-III depth and OpenCLIP classification. Object discovery with LOST gains 15.4 points for DeiT-III and 20.1 for DINOv2, and loses 1.7 for OpenCLIP.Appendix C: for OpenCLIP, LOST reads the attention values, and the value projection already suppresses the outliers in the model without registers, so removing them changes little. The results support a benefit to patch features, not a uniform gain on every task.

Questions and related work

Are registers the same as [CLS]?

Both start as learned vectors and are updated by every transformer block. Their outputs serve different roles: the class token is used as the global image representation, while the register outputs are discarded. Registers can also develop attention patterns different from [CLS] and from each other during training.

Why did the small models have no artifacts?

The paper does not establish a general size threshold. In its DINOv2 study, artifacts appear in ViT-L and larger models after enough training, while the tested ViT-S and ViT-B checkpoints stay clean. DeiT-III and OpenCLIP produce artifacts at ViT-B, so model size alone does not decide the outcome.

Does every vision transformer have them?

No. The original DINO shows none, and its attention maps were known for outlining objects without any labels. Neither does a ViT-L trained as a masked autoencoder (MAE). The authors hypothesize that MAE’s patch-reconstruction objective reduces the incentive to create these outliers. A patch-level loss still allows the model to use information from across the image.Paper, Fig. 2 and Appendix E. The MAE explanation is stated as a hypothesis.

Is this the same as attention sinks in language models?

They are related observations, but the papers do not establish one shared mechanism. StreamingLLM calls the early tokens that receive large attention scores “attention sinks”. Other work finds massive activations concentrated in a few token positions of language models. ViT registers address high-norm patch artifacts by adding learned token positions. All three involve unusual behavior at a small number of sequence positions.

Can registers be added to a model that is already trained?

The 2024 paper trains its models with learned registers from the start. Later work, Vision Transformers Don’t Need Trained Registers (2025), reports a training-free method that shifts the high-norm activations of an existing model into an added token. That method edits internal activations; inserting a blank token into an unchanged frozen model is a different thing and is not what either paper does.