ACL Findings2026
DiffER: Diffusion Entity-Relation Modeling for Reversal Curse in Diffusion Large Language Models
Shaokai He, Kaiwen Wei, Xinyi Zeng, Xiang Chen, Xue Yang, Zhenyang Li, Jiang Zhong, Yu Tian
Abstract
The "reversal curse" refers to the phenomenon where large language models (LLMs) exhibit predominantly unidirectional behavior when processing logically bidirectional relationships. Prior work attributed this to autoregressive training-predicting the next token inherently favors left-to-right information flow over genuine bidirectional knowledge associations. However, we observe that Diffusion LLMs (DLLMs), despite being trained bidirectionally, also suffer from the reversal curse. To investigate the root causes, we conduct systematic experiments on DLLMs and identify three key reasons: 1) entity fragmentation during training, 2) data asymmetry, and 3) missing entity relations. Motivated by the analysis of these reasons, we propose Diffusion Entity-Relation Modeling (DiffER), which addresses the reversal curse through entity-aware training and balanced data construction. Specifically, DiffER introduces whole-entity masking, which mitigates entity fragmentation by predicting complete entities in a single step. DiffER further employs distribution-symmetric and relation-enhanced data construction strategies to alleviate data asymmetry and missing relations. Extensive experiments demonstrate that DiffER effectively alleviates the reversal curse in Diffusion LLMs, offering new perspectives for future research. The code is available at https://github.com/ CQU-MM-Intelligent-Lab/DiffER.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Large Language Diffusion Models cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Large Language Diffusion Models cites Diffusion-LM Improves Controllable Text Generation
- Large Language Diffusion Models cites Simplified and Generalized Masked Diffusion for Discrete Data
- Diffusion-LM Improves Controllable Text Generation cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Simplified and Generalized Masked Diffusion for Discrete Data cites Structured Denoising Diffusion Models in Discrete State-Spaces
- Simplified and Generalized Masked Diffusion for Discrete Data cites Diffusion-LM Improves Controllable Text Generation
Builds on16
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- Structured Denoising Diffusion Models in Discrete State-SpacesJacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow et al.NeurIPS 2021 · 2,256 citations
- Diffusion-LM Improves Controllable Text GenerationXiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang et al.NeurIPS 2022 · 1,546 citations
- Large Language Diffusion ModelsShen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang et al.NeurIPS 2025 · 949 citations
- Simplified and Generalized Masked Diffusion for Discrete DataJiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet et al.NeurIPS 2024 · 693 citations
Related papers
- Rethinking the Reversal Curse of LLMs: a Prescription from Human Knowledge ReversalZhicong Lu, Li Jin, Peiguang Li, Yu Tian et al.EMNLP 2024 · 1 citation
- An Analysis and Mitigation of the Reversal CurseAng Lv, Kaiyi Zhang, Shufang Xie, Quan Tu et al.EMNLP 2024
- Breaking the Reversal Curse in Autoregressive Language Models via Identity BridgeXutao Ma, Yixiao Huang, Hanlin Zhu, Somayeh SojoudiICML 2026 · 2 citations
- Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation TrainingQingyan Guo, Rui Wang, Junliang Guo, Xu Tan et al.ACL Findings 2024 · 2 citations
- The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and MoreOuail Kitouni, Niklas Nolte, Adina Williams, Michael Rabbat et al.NeurIPS 2024 · 29 citations
