LLaDA 8B

Training by filling in blanks

A conventional, autoregressive model is trained to predict each token from the tokens before it. LLaDA is trained differently. It takes a training sentence, replaces a random selection of its tokens with a special mask token, and learns to predict the hidden tokens from all the visible ones, on both sides. The fraction hidden, t, changes from one training example to the next, from almost none to almost all. [1]

Take “The small robot reads a book.” Hide “robot” and “book”, and the words around each blank narrow down what fits. Hide almost everything, and the model has to propose a whole plausible sentence. Training at every masking level prepares the model for every stage of generation, from a reply that is all blanks to one that is nearly finished.

When t is small, an example has few blanks to learn from. The loss multiplies the error on the masked tokens by 1/t to compensate. This whole process, hiding tokens and learning to restore them, is called masked diffusion. The model itself is still a transformer; only the training objective and the way it generates text have changed.

Generating several tokens per round

To generate a reply, LLaDA keeps the prompt visible and fills the reply area with masks. In each round, the model predicts a token for every masked position, and a schedule decides which predictions to keep. With low-confidence remasking, the model keeps the predictions it is most sure of and turns the rest back into masks for the next round. With random remasking, the choice ignores confidence. The project demonstrates both. [2]

In the figure, press “Denoise one round” to reveal two positions at a time. The scattered order fills separated positions first, imitating a confidence-based schedule; the left-to-right order is there for comparison. The sentence and the order are scripted, so the scheduling idea can be seen on its own. A real model would produce probabilities for every masked position, and its confidence would set the order.

A later word can appear before an earlier one. That changes how a chat interface shows the reply: it cannot simply stream words left to right, and may need to show blanks or wait for a block to finish. In this illustration, a revealed word stays fixed. An answer that appears before its explanation does not show that the model worked out the answer first.

Why parallel prediction is not automatically faster

Predicting many positions in one pass sounds faster than producing one token per pass, but the full cost is less simple. An autoregressive model never revisits earlier tokens, so it can keep their keys and values in a KV cache and reuse them. In LLaDA, every position attends to every other, and each round changes some of them, so that cache no longer works.

The paper states the limitations plainly. Generation is slow, the reply length must be fixed in advance, and using fewer rounds lowers quality. The fixed length is less restrictive than it sounds, because the model can write an end-of-text marker and everything after it is dropped. [1] The original LLaDA could also generate block by block from left to right, filling each block in any order, a mode the paper calls semi-autoregressive. [1] The repository now links later work on variable-length generation and caching for diffusion models; those are separate advances. [3]

For a 128-token reply, eight rounds means eight passes through the model instead of 128, but each pass processes every position. Comparing 8 with 128 does not settle which approach is faster. Hardware use, prompt and reply length, attention cost, cache reuse and output quality all matter. A fair test runs both on the same hardware and compares replies of equal quality.

What LLaDA showed, and when it appeared

The model card lists Base and Instruct checkpoints, which need custom code to run. [4] The base model only fills in text. The Instruct model was trained further on conversations, still with the masked-diffusion objective.

The paper’s claim is narrower than “diffusion beats autoregression”. It shows that a diffusion model of this size can learn many of the abilities of a conventional language model. One result concerns reversal: given a line of a poem, LLaDA was better than GPT-4o at producing the line that comes before it. Left-to-right models tend to struggle in that direction, so the result says something about generation order. It is not a general measure of ability. [1]

The release date is uncertain. The official repository later labeled the work February 14, 2025, but the Instruct checkpoint’s history records its first commit containing weights on February 20. The page uses February 20 because that artifact can be verified; when the repository first became public cannot be reconstructed. [5] LLaDA 1.5, LLaDA-MoE and iLLaDA came later and are separate models.

Simplifications and limits

• The release date is the first recorded commit containing weights, not an independently verified public announcement.
• The sentence and unmasking order in the figure are scripted by hand, not produced by a model.
• The original LLaDA release did not include its full training data or training framework, only the weights and the code to run them.

Follow the idea

MambaBitNet b1.58 2B4T

Sources and further reading

Primary papers, model cards, and release posts. Facts describe the versions named here; the interactive calculations are teaching models. Research cutoff: October 3, 2026.

1. Large Language Diffusion ModelsSections 2.1–2.4: masking objective, bidirectional attention, SFT, and inference.
2. LLaDA project pageVisual generation examples and remasking strategies.
3. Official LLaDA repository and FAQOriginal speed limitations, public release notes, later variants, and license distinctions.
4. LLaDA-8B-Instruct model cardReleased model and inference setup.
5. LLaDA Instruct checkpoint historyFebruary 20, 2025 weight-bearing commit; staging/public-visibility uncertainty is stated.