Unmasking Text

An interactive guide to diffusion language models

My favorite architectural changes in deep learning are the ones that feel like a huge limitation has been lifted from the model. Convnets felt that way, and so did resnets. My latest favorite is diffusion for language models. A traditional LLM predicts one token at a time and never edits the past. Doesn't that seem wrong? How frustrated would you be if you were forced to write that way? At least give me a scratch space where I can noodle around. Diffusion is slowly giving models that scratch space, and it is exciting to watch it progress. This tutorial shows the progress so far.

Diffusion vs autoregression

Before the details, here is what token generation looks like for the three methods in this tutorial.

  1. Autoregressive: Like vanilla LLMs. The next token is generated one at a time.
  2. Masked diffusion: Many tokens are generated at the same time.
  3. Block diffusion: The text is cut into chunks A, B, C. Masked diffusion runs on A, then B, then C. Sort of a hybrid between 1 and 2.
Three ways to finish a sentencespeed

Autoregressive

Qwen2.5-7B-Instruct · one token per step
step 0

Masked diffusion

LLaDA-8B-Instruct · 16 steps · confidence order
step 0

Block diffusion

LLaDA-8B-Instruct · blocks of 8 · 64 steps
step 0

The masked and block panes are one model, LLaDA-8B-Instruct, run two ways. The autoregressive pane is Qwen2.5-7B-Instruct. Every trajectory on this page was recorded from those two models on a single RTX 3090. A few of the papers' results did not replicate on one card with a handful of prompts, but for the most part it did a pretty good job.

How to apply diffusion to language

If you are like me, you learned diffusion on images. All you do is take a noisy image and train a model to predict the original one. Diffusion progresses by taking tiny steps toward that predicted image. Over a series of many steps you can get very detailed results.

Below is a real run of a 2020 diffusion model trained on 126,000 photographs of churches. The left panel is what the network sees at step t. The right panel is its guess of the clean photo at that step. At step 550 the left panel still looks like static to a person, and the guess already has a steeple. The forward setting shows the finished photo with the noise of any step added in one jump, which is possible because each step's noise is Gaussian and Gaussians add.

Denoising a photographprocessstep t

What the network sees

the image at step t
the image at step t

Its guess of the clean photo

what the network predicts the noise is hiding
the predicted clean image

It works on images because pixels are numbers. Adding 0.1 to a pixel changes it a little. A token is not a number you can just nudge. Token 4,071 and token 4,072 are not neighbours in any sense that matters, and "a little noise" on a token id gives you a random other word.

In 2021 the D3PM paper kept the steps and changed what noise means. Each step, every token independently either stays put or gets replaced. Whether it is replaced is a coin flip whose odds rise as the steps go on. The rule is fixed in advance and never learned. What replaces the token is the design choice, and D3PM tried three:

  1. Uniform: A token drawn from the whole vocabulary.
  2. Nearest neighbour: A token drawn from the ones near it in embedding space. Words drift toward near-synonyms.
  3. Absorbing: A single catch-all [MASK] token, and once masked, masked for good.

The sentence below is corrupted live under each rule. The neighbours are indeed real examples from LLaDA's embedding table. Slide the noise to the middle and compare what you can and cannot recover.

Three ways to corrupt a sentencecorruptionnoise t 0.00

Run it backwards with the absorbing rule. Start with the prompt and sixteen [MASK] tokens. The network outputs a distribution over words for every blank, and with nothing to go on each one is broad and favours filler. Draw a word for three of the blanks and write them in. The network looks again. The thirteen distributions left are sharper, because a "lighthouse" three slots to the left changes what fits. Fill three more, look again, and stop when no blanks remain.

The image moved a little everywhere at every step. The sentence moves in jumps a few positions at a time. Once a token is filled it never moves again. What moves a little each step is the network's opinion about the blanks left. The sampler further down shows that opinion as ghost text inside each blank. The uniform rule, however, has no blanks. The response starts as sixteen random words, each step swaps a few of them, and a word can be swapped several times before it settles.

Masking also makes the mutations visible. A [MASK] says a word was here. A swapped word says nothing, and the model must work out whether it is being misled before it can fix anything. Almost every diffusion language model since 2021 uses masking so we will only see that for the rest of the tutorial.

Papers: D3PM (Austin et al., NeurIPS 2021). Background: DDPM (Ho et al., NeurIPS 2020).

What the network does

The diffusion network is a transformer with the causal mask removed. Every position attends to every other, in both directions. The input is the whole sequence at once, prompt and partly masked response, where [MASK] is an ordinary token with its own embedding. The output head is BERT's: at every position, a softmax over the vocabulary, trained so that a masked position predicts the word that was there. It has the same blocks and the same parameter count as a GPT of that size. The causal mask is the only thing missing.

Click any word below to mask it, and see the distribution the model puts on that one hole.

One hole, one distributionclick a word to mask it

One step of generation is one forward pass. The pass produces a distribution at every masked position at once, the sampler commits some of them, and the next pass sees a different sentence. (Note I will not cover it here but this means that the KV cache cannot be reusued, which is an important consideration for optimization.) During generation the weights never change. Only the input does, a few words at a time. Once the model has committed to a new token its priors snap on to its decision. This makes sense. If you pick a word it makes its neighboring words more certain. If you mask a word its neighbors become less certain.

In 2024 the SEDD paper showed that diffusion models could actually perform competitively with traditional LLMs. It recast the reverse process in terms of likelihood ratios, how much more probable a sentence becomes when one position changes, and gave a loss that learns those ratios stably along with a real likelihood bound. For the first time a diffusion language model could be scored against GPT-2 on perplexity even when diffusion executed in fewer steps.

Papers: SEDD (Lou, Meng and Ermon, ICML 2024, best paper).

Training is BERT with a random mask rate

Now that masked strategies had been embraced, MDLM and other papers helped to simplify the training procedure. First take a sentence from the corpus. Then pick a mask rate t at random between 0 and 1 and hide each token with that probability. A draw of t = 0.3 hides about three words in ten. Run the network on what is left. At each hidden position it gives a distribution over words, and the loss at that position is the cross-entropy against the word that was really there, the same loss BERT uses. Add those up over the hidden positions and multiply the total by 1/t. That multiplier is the only unusual part. A sentence with a tenth of its words hidden counts ten times as much as one with everything hidden.

The 2024 paper MDLM showed that this is not an approximation. Two rules, that the network never predicts [MASK] and that visible tokens are carried over unchanged, make the variational bound collapse to exactly that weighted cross-entropy. It is BERT's loss with the mask rate drawn at random instead of fixed at 15 percent. As a demonstration, they took a pretrained BERT off the shelf and fine-tuned it into a generator with this loss. A companion result, RADD, showed that the network does not even need to know t. LLaDA has no timestep input at all.

Below is one training step with predictions. Set the mask rate, reroll the mask, and click any masked slot to see the model's guesses and how much of the loss that slot contributes.

One training stepmask rate t 0.3click a masked slot

The carry-over rule has a price. Once the sampler commits a token, it is permanent. Commit a wrong word at step 3 of 32 and the remaining 29 steps build around the mistake.

Papers: MDLM (Sahoo et al., NeurIPS 2024). RADD (Ou et al., ICLR 2025) and MD4 (Shi et al., NeurIPS 2024) reached the same simplification concurrently.

Scaling up: LLaDA

MDLM's experiments were GPT-2 sized. LLaDA, in 2025, successfully trained the same objective at eight billion parameters on trillions of tokens.

To support question+response with a diffusion LLM all that was needed was to provide the question followed by a set of masked tokens. As the latter unmasks the answer is provided.

Pick a prompt. The prompt is pinned, and the response unmasks in front of you, at LLaDA's default settings.

Prompt, then unmaskpromptspeed

LLaDA-8B-Instruct

confidence order · 64 steps · blocks of 32
step 0

Qwen2.5-7B-Instruct

autoregressive, for comparison
step 0

LLaDA is competitive with LLaMA 3 of the same size on many benchmarks and behind on some, which the authors put down to less compute. The paper's most quoted result is a test it calls reversal poem completion. Given the end of a poem and asked for the lines before it, LLaDA does well and strong autoregressive models do badly. A model that only ever practiced predicting the next word has no way to run backwards. A masked model has practiced predicting any position from any context.

(Note: we tried to reproduce that result here and it did not work : ). In fact Qwen worked much better as you can see. It works on later experiments though.)

Papers: LLaDA (Nie et al., NeurIPS 2025). Dream 7B (Ye et al., 2025), Mercury (Inception Labs, 2025).

Making choices more confidently

In 2025 Kim and colleagues published a paper called Train for the Worst, Plan for the Best. Its premise is that the training objective from the last section asks for wildly different things depending on the mask. Fill in one missing word and the task is trivial. Reconstruct a sentence from its last word alone and the task is close to impossible. A Sudoku with three cells filled is the same kind of thing. The model trains on all of these patterns, weighted by the schedule, whether or not it will ever meet them.

Generation is where the choice comes in. You start fully masked and unmask a few positions per step, and nothing in the objective says which positions. The model was trained for the worst case. At generation you can plan for the best: commit the positions that are easy now and leave the hard ones until more of the sentence is in place.

That planning happens through three knobs an autoregressive model does not have.

How many steps: Sixty-four steps for sixty-four tokens means one token per step, each decided with the most context possible. Four steps means sixteen tokens per step, each decided with almost none.

Which positions: Random is the naive choice. LLaDA's alternative commits the positions where the model is most confident. Kim and colleagues analyse that rule and propose a variant that looks at the gap between the top two guesses instead of the top one. LLaDA calls its rule low-confidence remasking, meaning keep the unsure positions masked.

How many per step: Instead of a fixed quota, commit every position whose confidence clears a threshold. Easy stretches of text then go many tokens at a time and hard ones go slowly. The rule is from Fast-dLLM, and it is how deployed systems get their speed.

All three are below. The shading is confidence, and the ghost text in a blank is what the model would write there if forced to commit now.

The samplerpromptspeed stepsorderthresholdconfidence

LLaDA-8B-Instruct

masked diffusion · ghost text is the current top guess
step 0

Try this on the lighthouse prompt. At 4 steps, random order writes a decent opening and then commits an end-of-text token mid-sentence. Confidence order fills the page with the word the, because with nothing else on the page the most confident guess for almost every blank is the same word, and sixteen blanks committed at once cannot see each other. At 16 steps, order decides whether you get a paragraph or a stutter. At 64 steps, one token per step, all three orders produce a readable paragraph, and the choice of order stops mattering. The threshold rule at 0.5 finishes this prompt in 46 steps and the arithmetic prompt in 4.

Kim and colleagues also trained a masked model on Sudoku. With random order it solves a small fraction of puzzles. With confidence order, fill in the cells you are sure of first, the same model solves most of them. Nothing about the network changed.

Papers: Kim et al., Train for the Worst, Plan for the Best (ICML 2025). Fast-dLLM (Wu et al., 2025).

Blocks

Two practical problems remain. You must pick the length up front, which is why some trajectories above end in a run of blank end-of-text tokens. And every step re-attends over the whole sequence, with no cache, because everything might still change.

Block diffusion, in 2025, fixes both with one idea: cut the output into blocks of B tokens. Generate the blocks left to right, and the tokens inside each block by masked diffusion, conditioned on the finished blocks. A finished block never changes. It can be cached exactly as in an autoregressive model, and generation stops when a block contains the end token. Training uses a specially built attention mask so that all blocks train in one pass while seeing only what they would see at generation time. The paper has a diagram of that mask.

B is a dial. B = 1 is an autoregressive model. B = the whole length is LLaDA. The dial below runs the same model from B = 4 to 64. Watch the finished blocks turn gold.

The block-size dialpromptspeed block size Bsteps

LLaDA-8B-Instruct

confidence order within each block · finished blocks underlined gold
step 0

Perplexity improves steadily as blocks shrink toward autoregression. The paper also finds that much of the diffusion-side gap is training noise from sampling the mask rate, and that clipping the rate away from 0 and 1 narrows it. Some of the gap between the two families is an optimization artifact.

This is the architecture deployed systems use, and it is why the third pane at the top exists.

Papers: Block Diffusion (Arriola et al., ICLR 2025).

Taking a token back

Every model above commits permanently. For an autoregressive model that is a fact of life. For a model that could in principle revisit any position, it is a self-imposed limit, and removing it is the current frontier. This is where things stood in June 2026.

Training-free remasking changes only the sampler. ReMDM gives committed tokens some probability of being masked again each step, and other methods pick what to remask by the model's own confidence. A June 2026 re-evaluation found that under the short blocks deployed systems use, this adds little over confidence ordering alone. The gains appear with long blocks, where many tokens were committed on thin context. Trained self-correction goes further. RemeDi trains a per-token confidence to flag wrong tokens and remasks them, and another line treats the model's own mistakes as one more kind of noise and trains a corrector to undo them. LLaDA2.1 shipped token-to-token editing, and a follow-up argued for resetting suspect tokens to [MASK] instead of overwriting them. Two papers from June 2026 let later blocks reach back and revise earlier ones. What signal tells a model which of its tokens to doubt is where the effort is.

The replay below is the simplest training-free rule, run on LLaDA: when the model's probability for a committed token drops below 0.3 under the current context, the token goes back to [MASK]. On the poem-reversal prompt it changes the answer. The plain run ended both lines with shore, and here the first shore is taken back four times before the model settles on sea. On other prompts the rule mostly oscillates, taking back a comma and putting it straight back, and on two prompts it never fires. Both are typical of confidence-triggered remasking.

Remasking replaypromptspeed

LLaDA-8B-Instruct

confidence order · 64 steps · a committed token is remasked when its probability falls below 0.3
step 0

This is the frontier as I know it. My guess is that this kind of iteration will lead to a much higher quality of writing by LLMs in the future.

Papers: ReMDM (Wang et al., 2025). Frković, Jazbec et al. (arXiv 2606.12232). RemeDi (Huang et al., ICLR 2026). Schiff et al., Learn from Your Mistakes (2026). LLaDA2.1 (Bie et al., 2026). Yao, Targeted Remasking (2026).

Where to go next