Iterative Entropy-Guided Refinement: A post-hoc correction mechanism for Masked Diffusion Models that selectively re-masks and re-generates high-uncertainty tokens.

Motivation

While the paper optimizes the decoding path order, it assumes a single pass is sufficient. However, even with an optimal path, the model may generate inconsistent tokens in high-entropy regions. A second refinement pass targeting only the most uncertain tokens could significantly improve consistency without the cost of full autoregressive regeneration.

Proposed Method

Implement a two-stage decoding process. First, perform the standard decoding using the paper's optimized path. Second, calculate the Denoising Entropy for all generated tokens. Select the top-k% tokens with the highest residual entropy, re-mask them, and perform a short secondary diffusion process conditioned on the low-entropy (confident) tokens to harmonize the output.

Expected Contribution

A method to boost the fidelity and coherence of MDMs (specifically in image synthesis or long-text generation) with negligible computational overhead compared to full re-sampling.

Required Resources

Pre-trained Masked Diffusion Model (e.g., MaskGIT or MDLM), inference GPU compute.

Source Paper

Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty

View Paper Details →