Must Language Models Generate in Order? My Take on dLLMs

When I wrote about training a Transformer from scratch and taking MicroGPT apart, I followed a familiar path: given a prefix, predict the next token, then feed the result back into the input.

It feels natural. Chat interfaces reveal text a little at a time, and it becomes easy to treat “language model” and “left-to-right generation” as the same thing.

Diffusion language models made me separate those ideas again. The question that interests me is: if an answer can take shape across a set of unknown positions instead of being written in order, what changes in the generation system?

Here, dLLM means diffusion large language model. This article is my technical reading of public papers and implementations, not a local benchmark report. The diagrams, examples, and latency arithmetic explain mechanisms. The sections on code, agents, and Macs describe how I would evaluate this approach in engineering practice.

Separate the network from the generation process

A Transformer determines how information moves between positions. Autoregression describes how conditional probabilities organize sequence generation. These are different layers.

“Will diffusion replace the Transformer?” therefore asks the wrong question. LLaDA itself uses a Transformer; the main changes concern the training objective, attention visibility, and sampling procedure. The official LLaDA explanation explicitly separates these concepts.

I find it useful to divide a generation system into four layers:

The speed of text appearing on screen tells me little about which layer improved. Character-by-character display does not prove token-by-token computation, and a paragraph appearing at once does not prove diffusion.

From continuing a prefix to filling unknown positions

Ordinary autoregressive decoding can be summarized as:

Prefix → predict one token → append it → predict again

This describes generation. During training, the full example is known and predictions at multiple positions can be computed in parallel. Speculative decoding also lets an autoregressive model verify several candidates together, reducing serial calls. “Autoregression cannot parallelize at all” is therefore inaccurate. The speculative decoding paper describes this acceleration route.

Masked diffusion organizes the process differently: leave positions unknown, then infer missing content from what is currently visible. The following uses word chunks for illustration; the brackets are not an actual tokenizer trace:

Initial    [MASK] [MASK]  [MASK]     [MASK]
Round 1    [MASK] [model] [MASK]     [answers]
Round 2    [This] [model] [MASK]     [answers]
Round 3    [This] [model] [generates] [answers]

The point is that the order in which positions become settled can differ from text order. This does not promise completion in three rounds, or acceptance of every candidate predicted in a round.

LLaDA trains on inputs with varying amounts of random masking and learns to recover missing tokens. Generation starts from a masked state and progressively resolves the output. This is discrete masked diffusion. It shares the idea of iterative denoising with image diffusion, but text tokens should not be mistaken for pixels corrupted by Gaussian noise. LLaDA paper

Candidates and accepted results also need to be distinguished. A prediction can be discarded and reconsidered in a later round. Whether an already accepted position can change depends on the sampler's remasking rules. Some processes only reveal previously unknown positions. Iteration does not mean every generated token can always be rewritten.

How I read LLaDA and Mercury

LLaDA is a public entry point into this approach: an 8B masked diffusion model trained from scratch, with readable inference code. Its value to me is being able to connect the sampling procedure to an implementation instead of forming an opinion from a chat demo alone. Official repository

Mercury represents a commercial direction. Inception's initial announcement emphasized the speed potential of diffusion generation and highlighted coding as an important use case. Vendor performance claims need to be read alongside hardware, workloads, and baselines; they cannot be translated directly into gains on my own computer. Mercury launch announcement

I discuss them together to distinguish research evidence from product delivery. I am not assuming that Mercury internally follows LLaDA's specific sampler, and this is not a ranking of the latest model versions.

Why parallel generation is not necessarily faster

My main question is not how many tokens a model can predict at once. It is how long a usable answer takes.

A deliberately simplified model helps build intuition. Let N be output length and S the number of diffusion rounds:

Ordinary AR latency ≈ prompt prefill + N × average decode-step time
Diffusion latency   ≈ initialization + S × average denoising-round time
                      + scheduling and output overhead

This is not a complexity proof, and it leaves out optimizations such as speculative decoding and batching. It makes one point: S being smaller than N does not establish a speedup, because a denoising round and an incremental decode step have different costs.

Suppose an imaginary task needs 256 output tokens, with autoregressive decoding averaging 4 ms per step. The decoding portion takes about 1,024 ms. Diffusion with 32 rounds at 40 ms each takes 1,280 ms; at 16 rounds, it takes 640 ms. Every number here is a teaching assumption, not a measurement of any model or device.

Even after the number falls, another question remains: is the 16-round answer as usable as the 32-round answer? More failures and retries can spend the time just saved.

Cache reuse has conditions

In ordinary causal decoding, appending a token does not change old prefix representations, so their per-layer keys and values can be reused. With bidirectional denoising, changing other positions can change a position's context. Reusing the same cache without adjustment can mean using stale representations.

That does not make caching permanently unavailable to dLLMs. Fast-dLLM studies approximate KV caching and confidence-aware parallel decoding. Block Diffusion keeps autoregressive relationships between blocks and uses diffusion within each block, offering another design for caching and variable-length generation.

For an implementation, I would ask which positions are cached, when they are refreshed, and how approximation error is evaluated. “Uses KV cache” is only the beginning.

Simultaneous predictions are not necessarily jointly correct

Two positions can each look confident while being inconsistent together. Variable declarations and references, or JSON fields and values, have mutual constraints. Fast-dLLM discusses broken token dependencies during parallel decoding, making parallel acceptance a quality-control decision too. Paper

I would not translate “multiple positions are visible” into “the structure is automatically correct.” Training, sampling, and validation still separate seeing a constraint from satisfying it.

For code and agents, I care about the commit boundary

For code generation, I would start with function-body completion and local repair: fix the inputs, signature, and context, generate within a limited scope, then use a compiler and tests. That experiment makes failures easier to locate than generating an entire repository immediately.

For JSON, SQL, and my own HHY DSL, I would add schema validation, a parser, or type checking as appropriate. Diffusion offers another way to construct candidates. Validity still needs an independent check.

Agents add another question: when may generated content be executed?

Candidate arguments → settled model output → parsing and validation
→ permission checks → tool execution

I would treat intermediate candidates as drafts. An amount, file path, or SQL condition temporarily appearing on screen must not trigger a tool. A sampler accepting a token is an internal event; the application needs its own commit boundary.

The same rule applies to autoregressive agents. dLLMs simply make the distinction more visible to me: displayable text, stable output, and an action authorized for execution are three different states.

How I would evaluate it locally

My previous MLX experiments taught me to verify the complete pipeline before discussing speed. The following is my proposed dLLM evaluation plan; I have not run it for this article.

I would prepare three fixed task sets: function completion with tests, JSON generation with schemas, and DSL conversion with parsing and semantic checks. I would fix the prompt set and output budget, record model version, precision, device, context length, denoising rounds, block size, and cache configuration, and separate cold-start requests from warmed-up runs.

I would report both successes and failures so that selecting only successful samples does not create a speed illusion. Output tokens per second can remain in the report, but cannot replace these metrics.

The comparison would have two stages. First, vary sampling settings within the same diffusion model to map the relationship between speed and quality. Then compare task outcomes against an autoregressive model suited to the machine. The second comparison also includes differences in training data and model capability; those cannot all be attributed to the generation architecture.

On a Mac, fitting the weights is only the first hurdle. Operator support, quantization, memory bandwidth, cache layout, and sampler implementation all need validation. The names Ollama or MLX do not establish compatibility with a particular model.

Likewise, removing a GPT or Qwen model's causal mask does not automatically produce a usable diffusion model. Even when starting from existing weights, the training objective, sampling method, and evaluation process still need to match the new approach.

Where I stand

I want to keep following dLLMs because they reopen a choice that has become easy to take for granted: must language generation advance along a single prefix?

In a product, I ultimately need to deliver checked code, parseable structures, and reliably completed tasks. Predicting more positions in parallel is valuable. It becomes a reason to change an existing system when it produces repeatable gains in quality and latency.

My next step would be a small controlled experiment: the same function-completion tasks, with pass rate, complete latency, memory, and retry cost recorded. I want to find the boundary where it actually helps, then decide where it belongs in the workflow.

References

• Large Language Diffusion Models — LLaDA: masked diffusion language modeling and experiments.
• Official LLaDA implementation and FAQ: inference code, network structure, and efficiency limitations.
• Introducing Mercury — Inception: the initial commercial product announcement.
• Fast-dLLM: approximate caching, parallel decoding, and quality tradeoffs.
• Block Diffusion: autoregression between blocks and diffusion within blocks.
• Fast Inference from Transformers via Speculative Decoding: another route to reducing serial decoding calls in autoregressive models.