Unmasking Text
An interactive guide to diffusion language models
My favorite architectural changes in deep learning are the ones that feel like a huge limitation has been lifted from the model. Convnets felt that way, and so did resnets. My latest favorite is diffusion for language models. A traditional LLM predicts one token at a time and never edits the past. Doesn't that seem wrong? How frustrated would you be if you were forced to write that way? At least give me a scratch space where I can noodle around. Diffusion is slowly giving models that scratch space, and it is exciting to watch it progress. This tutorial shows the progress so far.
Diffusion vs autoregression
Before the details, here is what token generation looks like for the three methods in this tutorial.
- Autoregressive: Like vanilla LLMs. The next token is generated one at a time.
- Masked diffusion: Many tokens are generated at the same time.
- Block diffusion: The text is cut into chunks A, B, C. Masked diffusion runs on A, then B, then C. Sort of a hybrid between 1 and 2.
Autoregressive
Qwen2.5-7B-Instruct · one token per stepMasked diffusion
LLaDA-8B-Instruct · 16 steps · confidence orderBlock diffusion
LLaDA-8B-Instruct · blocks of 8 · 64 stepsThe masked and block panes are one model, LLaDA-8B-Instruct, run two ways. The autoregressive pane is Qwen2.5-7B-Instruct. Every trajectory on this page was recorded from those two models on a single RTX 3090. A few of the papers' results did not replicate on one card with a handful of prompts, but for the most part it did a pretty good job.
How to apply diffusion to language
If you are like me, you learned diffusion on images. All you do is take a noisy image and train a model to predict the original one. Diffusion progresses by taking tiny steps toward that predicted image. Over a series of many steps you can get very detailed results.
Below is a real run of a 2020 diffusion model trained on 126,000 photographs of churches. The left panel is what the network sees at step t. The right panel is its guess of the clean photo at that step. At step 550 the left panel still looks like static to a person, and the guess already has a steeple. The forward setting shows the finished photo with the noise of any step added in one jump, which is possible because each step's noise is Gaussian and Gaussians add.
What the network sees
the image at step tIts guess of the clean photo
what the network predicts the noise is hidingIt works on images because pixels are numbers. Adding 0.1 to a pixel changes it a little. A token is not a number you can just nudge. Token 4,071 and token 4,072 are not neighbours in any sense that matters, and "a little noise" on a token id gives you a random other word.
In 2021 the D3PM paper kept the steps and changed what noise means. Each step, every token independently either stays put or gets replaced. Whether it is replaced is a coin flip whose odds rise as the steps go on. The rule is fixed in advance and never learned. What replaces the token is the design choice, and D3PM tried three:
- Uniform: A token drawn from the whole vocabulary.
- Nearest neighbour: A token drawn from the ones near it in embedding space. Words drift toward near-synonyms.
- Absorbing: A single catch-all
[MASK]token, and once masked, masked for good.
The sentence below is corrupted live under each rule. The neighbours are indeed real examples from LLaDA's embedding table. Slide the noise to the middle and compare what you can and cannot recover.
Run it backwards with the absorbing rule. Start with the prompt and sixteen [MASK] tokens. The network outputs a distribution over words for every blank, and with nothing to go on each one is broad and favours filler. Draw a word for three of the blanks and write them in. The network looks again. The thirteen distributions left are sharper, because a "lighthouse" three slots to the left changes what fits. Fill three more, look again, and stop when no blanks remain.
The image moved a little everywhere at every step. The sentence moves in jumps a few positions at a time. Once a token is filled it never moves again. What moves a little each step is the network's opinion about the blanks left. The sampler further down shows that opinion as ghost text inside each blank. The uniform rule, however, has no blanks. The response starts as sixteen random words, each step swaps a few of them, and a word can be swapped several times before it settles.
Masking also makes the mutations visible. A [MASK] says a word was here. A swapped word says nothing, and the model must work out whether it is being misled before it can fix anything. Almost every diffusion language model since 2021 uses masking so we will only see that for the rest of the tutorial.
Papers: D3PM (Austin et al., NeurIPS 2021). Background: DDPM (Ho et al., NeurIPS 2020).
What the network does
The diffusion network is a transformer with the causal mask removed. Every position attends to every other, in both directions. The input is the whole sequence at once, prompt and partly masked response, where [MASK] is an ordinary token with its own embedding. The output head is BERT's: at every position, a softmax over the vocabulary, trained so that a masked position predicts the word that was there. It has the same blocks and the same parameter count as a GPT of that size. The causal mask is the only thing missing.
Click any word below to mask it, and see the distribution the model puts on that one hole.
One step of generation is one forward pass. The pass produces a distribution at every masked position at once, the sampler commits some of them, and the next pass sees a different sentence. (Note I will not cover it here but this means that the KV cache cannot be reusued, which is an important consideration for optimization.) During generation the weights never change. Only the input does, a few words at a time. Once the model has committed to a new token its priors snap on to its decision. This makes sense. If you pick a word it makes its neighboring words more certain. If you mask a word its neighbors become less certain.
In 2024 the SEDD paper showed that diffusion models could actually perform competitively with traditional LLMs. It recast the reverse process in terms of likelihood ratios, how much more probable a sentence becomes when one position changes, and gave a loss that learns those ratios stably along with a real likelihood bound. For the first time a diffusion language model could be scored against GPT-2 on perplexity even when diffusion executed in fewer steps.
Papers: SEDD (Lou, Meng and Ermon, ICML 2024, best paper).
Training is BERT with a random mask rate
Now that masked strategies had been embraced, MDLM and other papers helped to simplify the training procedure. First take a sentence from the corpus. Then pick a mask rate t at random between 0 and 1 and hide each token with that probability. A draw of t = 0.3 hides about three words in ten. Run the network on what is left. At each hidden position it gives a distribution over words, and the loss at that position is the cross-entropy against the word that was really there, the same loss BERT uses. Add those up over the hidden positions and multiply the total by 1/t. That multiplier is the only unusual part. A sentence with a tenth of its words hidden counts ten times as much as one with everything hidden.
The 2024 paper MDLM showed that this is not an approximation. Two rules, that the network never predicts [MASK] and that visible tokens are carried over unchanged, make the variational bound collapse to exactly that weighted cross-entropy. It is BERT's loss with the mask rate drawn at random instead of fixed at 15 percent. As a demonstration, they took a pretrained BERT off the shelf and fine-tuned it into a generator with this loss. A companion result, RADD, showed that the network does not even need to know t. LLaDA has no timestep input at all.
Below is one training step with predictions. Set the mask rate, reroll the mask, and click any masked slot to see the model's guesses and how much of the loss that slot contributes.
The carry-over rule has a price. Once the sampler commits a token, it is permanent. Commit a wrong word at step 3 of 32 and the remaining 29 steps build around the mistake.
Papers: MDLM (Sahoo et al., NeurIPS 2024). RADD (Ou et al., ICLR 2025) and MD4 (Shi et al., NeurIPS 2024) reached the same simplification concurrently.
Scaling up: LLaDA
MDLM's experiments were GPT-2 sized. LLaDA, in 2025, successfully trained the same objective at eight billion parameters on trillions of tokens.
To support question+response with a diffusion LLM all that was needed was to provide the question followed by a set of masked tokens. As the latter unmasks the answer is provided.
Pick a prompt. The prompt is pinned, and the response unmasks in front of you, at LLaDA's default settings.
LLaDA-8B-Instruct
confidence order · 64 steps · blocks of 32Qwen2.5-7B-Instruct
autoregressive, for comparisonLLaDA is competitive with LLaMA 3 of the same size on many benchmarks and behind on some, which the authors put down to less compute. The paper's most quoted result is a test it calls reversal poem completion. Given the end of a poem and asked for the lines before it, LLaDA does well and strong autoregressive models do badly. A model that only ever practiced predicting the next word has no way to run backwards. A masked model has practiced predicting any position from any context.
(Note: we tried to reproduce that result here and it did not work : ). In fact Qwen worked much better as you can see. It works on later experiments though.)
Papers: LLaDA (Nie et al., NeurIPS 2025). Dream 7B (Ye et al., 2025), Mercury (Inception Labs, 2025).
Making choices more confidently
In 2025 Kim and colleagues published a paper called Train for the Worst, Plan for the Best. Its premise is that the training objective from the last section asks for wildly different things depending on the mask. Fill in one missing word and the task is trivial. Reconstruct a sentence from its last word alone and the task is close to impossible. A Sudoku with three cells filled is the same kind of thing. The model trains on all of these patterns, weighted by the schedule, whether or not it will ever meet them.
Generation is where the choice comes in. You start fully masked and unmask a few positions per step, and nothing in the objective says which positions. The model was trained for the worst case. At generation you can plan for the best: commit the positions that are easy now and leave the hard ones until more of the sentence is in place.
That planning happens through three knobs an autoregressive model does not have.
How many steps: Sixty-four steps for sixty-four tokens means one token per step, each decided with the most context possible. Four steps means sixteen tokens per step, each decided with almost none.
Which positions: Random is the naive choice. LLaDA's alternative commits the positions where the model is most confident. Kim and colleagues analyse that rule and propose a variant that looks at the gap between the top two guesses instead of the top one. LLaDA calls its rule low-confidence remasking, meaning keep the unsure positions masked.
How many per step: Instead of a fixed quota, commit every position whose confidence clears a threshold. Easy stretches of text then go many tokens at a time and hard ones go slowly. The rule is from Fast-dLLM, and it is how deployed systems get their speed.
All three are below. The shading is confidence, and the ghost text in a blank is what the model would write there if forced to commit now.
LLaDA-8B-Instruct
masked diffusion · ghost text is the current top guessQwen2.5-7B-Instruct
autoregressiveTry this on the lighthouse prompt. At 4 steps, random order writes a decent opening and then commits an end-of-text token mid-sentence. Confidence order fills the page with the word the, because with nothing else on the page the most confident guess for almost every blank is the same word, and sixteen blanks committed at once cannot see each other. At 16 steps, order decides whether you get a paragraph or a stutter. At 64 steps, one token per step, all three orders produce a readable paragraph, and the choice of order stops mattering. The threshold rule at 0.5 finishes this prompt in 46 steps and the arithmetic prompt in 4.
Kim and colleagues also trained a masked model on Sudoku. With random order it solves a small fraction of puzzles. With confidence order, fill in the cells you are sure of first, the same model solves most of them. Nothing about the network changed.
Papers: Kim et al., Train for the Worst, Plan for the Best (ICML 2025). Fast-dLLM (Wu et al., 2025).
Blocks
Two practical problems remain. You must pick the length up front, which is why some trajectories above end in a run of blank end-of-text tokens. And every step re-attends over the whole sequence, with no cache, because everything might still change.
Block diffusion, in 2025, fixes both with one idea: cut the output into blocks of B tokens. Generate the blocks left to right, and the tokens inside each block by masked diffusion, conditioned on the finished blocks. A finished block never changes. It can be cached exactly as in an autoregressive model, and generation stops when a block contains the end token. Training uses a specially built attention mask so that all blocks train in one pass while seeing only what they would see at generation time. The paper has a diagram of that mask.
B is a dial. B = 1 is an autoregressive model. B = the whole length is LLaDA. The dial below runs the same model from B = 4 to 64. Watch the finished blocks turn gold.
LLaDA-8B-Instruct
confidence order within each block · finished blocks underlined goldPerplexity improves steadily as blocks shrink toward autoregression. The paper also finds that much of the diffusion-side gap is training noise from sampling the mask rate, and that clipping the rate away from 0 and 1 narrows it. Some of the gap between the two families is an optimization artifact.
This is the architecture deployed systems use, and it is why the third pane at the top exists.
Papers: Block Diffusion (Arriola et al., ICLR 2025).
Taking a token back
Every model above commits permanently. For an autoregressive model that is a fact of life. For a model that could in principle revisit any position, it is a self-imposed limit, and removing it is the current frontier. This is where things stood in June 2026.
Training-free remasking changes only the sampler. ReMDM gives committed tokens some probability of being masked again each step, and other methods pick what to remask by the model's own confidence. A June 2026 re-evaluation found that under the short blocks deployed systems use, this adds little over confidence ordering alone. The gains appear with long blocks, where many tokens were committed on thin context. Trained self-correction goes further. RemeDi trains a per-token confidence to flag wrong tokens and remasks them, and another line treats the model's own mistakes as one more kind of noise and trains a corrector to undo them. LLaDA2.1 shipped token-to-token editing, and a follow-up argued for resetting suspect tokens to [MASK] instead of overwriting them. Two papers from June 2026 let later blocks reach back and revise earlier ones. What signal tells a model which of its tokens to doubt is where the effort is.
The replay below is the simplest training-free rule, run on LLaDA: when the model's probability for a committed token drops below 0.3 under the current context, the token goes back to [MASK]. On the poem-reversal prompt it changes the answer. The plain run ended both lines with shore, and here the first shore is taken back four times before the model settles on sea. On other prompts the rule mostly oscillates, taking back a comma and putting it straight back, and on two prompts it never fires. Both are typical of confidence-triggered remasking.
LLaDA-8B-Instruct
confidence order · 64 steps · a committed token is remasked when its probability falls below 0.3This is the frontier as I know it. My guess is that this kind of iteration will lead to a much higher quality of writing by LLMs in the future.
Papers: ReMDM (Wang et al., 2025). Frković, Jazbec et al. (arXiv 2606.12232). RemeDi (Huang et al., ICLR 2026). Schiff et al., Learn from Your Mistakes (2026). LLaDA2.1 (Bie et al., 2026). Yao, Targeted Remasking (2026).
Where to go next
- The NeurIPS 2026 workshops DiffuLM and Beyond Next Token Prediction.
- A Survey on Diffusion Language Models (arXiv 2508.10875) and its Awesome-DLMs list, which tracks new papers by date.
- Accelerating Masked Diffusion LLMs (arXiv 2607.12829), on when parallel decoding actually saves time.