Conditional [MASK] Discrete Diffusion Language Model

Although auto-regressive models excel in natural language processing, they often struggle to generate diverse text and provide limited controllability. Non-auto-regressive methods could be an alternative but often produce degenerate outputs and exhibit shortcomings in conditional generation. To address these challenges, we propose Diffusion-EAGS, a novel framework that integrates conditional masked language models into diffusion language models through the theoretical lens of a conditional Markov Random Field. In doing so, we propose entropy-adaptive Gibbs sampling and entropybased noise scheduling to counterbalance each model's shortcomings. Experimental results show that Diffusion-EAGS outperforms baselines and achieves the best quality-diversity tradeoff, demonstrating its effectiveness in nonautoregressive text generation. Context Jake was playing with his toys. He accidentally broke his favorite one. He cried a lot over it. His parents decided to replace it for him. Keyword not stop Jake just could not stop crying. Jake feel It made Jake feel So much better. would enjoy Jake said he would enjoy the new toy Context Neil was in Sofia, Bulgaria. He was enjoying a trip backpacking through Europe. ... He thought the food and culture in Sofia were the best. Keyword Bulgaria! Things were looking great in Bulgaria! Context Karen wanted to go on a trip to France. She started doing research on the trip. She decided to book a week long trip. She left the next day for her trip. Keyword her trip She spent almost a week there during her trip.