From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

Researchers show that any-order training alone does not guarantee any-order inference in masked diffusion language models because of positional uncertainty.

They propose insertion-based masked diffusion and latent-space masked diffusion to better support non-linear generation.

The methods aim to let models reason and generate in flexible order, which is useful for code generation and math solving.

Reported results suggest improvements on tasks such as GSM8K and Python coding.