Gumbel Distillation for Parallel Text Generation

The slow, sequential nature of autoregressive (AR) language models has driven the adoption of parallel decoding methods. However, these non-AR models often sacrifice generation quality as they struggle to model the complex joint distribution of token sequences. To narrow this performance gap, we introduce Gumbel Distillation, a novel distillation technique that enables parallel decoders to learn this distribution effectively. Our method leverages the Gumbel-Max trick to create a deterministic mapping from a latent Gumbel noise space to the output tokens of a high-performing AR teacher. As a model-agnostic technique, Gumbel Distillation seamlessly integrates with diverse parallel decoding architectures, including MDLM and BD3-LM. Experiments on LM1B and OpenWebText show that Gumbel Distillation substantially improves the generation quality of parallel language models, achieving a 30.0% improvement in MAUVE score and 10.5% in generative perplexity over MDLM trained on OpenWebText dataset. Code available at: https://github.com/hxixixh/gumbel-distill . INTRODUCTION Autoregressive (AR) language models have set the standard for text generation (Radford et al., 2019; Brown et al., 2020; Touvron et al., 2023) , but their sequential, token-by-token inference process introduces significant latency, hindering their use in real-world applications. To address this bottleneck, various parallel decoding methods have emerged, including Masked Diffusion Language Models (MDLMs) (