Imagine a whole git repo of a project materializing like that.

This animation is a perfect abstract!

Those early tokens aren't necessarily immutable, they still could be "edited" depending on UI. Human conversation and even internal compositional cogitation is full of "what I meant by that" or "on second thought" type clarifications and corrections. Sometimes these aren't verbosely disclaimed, there's body language involved. Likewise there could be occasional lookback parsing and later blocks could convey modifications. The UI can then highlight those revisions transparently by applying strikethrough styling, coloration, dotted underline with tooltip on hover, etc.

Like we've seen with human interactions and media, this may be susceptible to misinterpretation by the reader or listener, especially via second-hand clips or screenshots lacking full context. But if the UX is clean and speedy it would be less likely.

To be fair, it's not "obviously" better, but it opens a new point on the tradeoff curve. For a lot of use cases full autoregression is clearly better, and for some others ful diffusion will still be better.

Autoregressivity has high quality outputs but is fairly slow. Diffusion has low quality output but is quite fast.

This allows you to go in the middle, not as high quality as full autoregression and not as fast as full diffusion, but a balance between both.

the LLaDA paper is a scaled-up version of this paper; they cite it as an anonymous ICLR submission

For LLM, standard method to provide to foundation model sentence without one word (random, but already known to model) and ask to fill this word.

Not familiar at all with diffusion LLM but I'd guess you'd have noisy logits.

Could you expound on this? From what I'm reading, this sounds like an issue with diffusion models that their block diffusion model is purposefully designed to mitigate, by conditioning on previous blocks and allowing for larger blocks if that conditioning still doesn't help maintain coherence.

My understanding here is block size can be arbitrarily large, under similar constraints as diffusion models. Is that not the case?

Excellent question