EMNLP Findings2020
Revisiting Representation Degeneration Problem in Language Modeling
Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao, Qinli Yang, Junming Shao
Abstract
Weight tying is now a common setting in many language generation tasks such as language modeling and machine translation. However, a recent study reveals that there is a potential flaw in weight tying. They find that the learned word embeddings are likely to degenerate and lie in a narrow cone when training a language model. They call it the representation degeneration problem and propose a cosine regularization to solve it. Nevertheless, we prove that the cosine regularization is insufficient to solve the problem, as the degeneration is still likely to happen under certain conditions. In this paper, we revisit the representation degeneration problem and theoretically analyze the limitations of the previously proposed solution. Afterward, we propose an alternative regularization method called Laplacian regularization to tackle the problem. Experiments on language modeling demonstrate the effectiveness of the proposed Laplacian regularization.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings cites Improving Neural Language Generation with Spectrum Control
- Better Embeddings with Coupled Adam cites Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings
- Better Embeddings with Coupled Adam cites Stable Anisotropic Regularization
- Better Embeddings with Coupled Adam cites Improving Neural Language Generation with Spectrum Control
Cited by top-tier papers9
- Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token EmbeddingsSangwon Yu, Jongyoon Song, Heeseung Kim, Seongmin Lee et al.ACL 2022 · 42 citations
- Stable Anisotropic RegularizationWilliam Rudman, Carsten EickhoffICLR 2024 · 13 citations
- ReSel: N-ary Relation Extraction from Scientific Text and Tables by Learning to Retrieve and SelectYuchen Zhuang, Yinghao Li, Junyang Zhang, Yue Yu et al.EMNLP 2022 · 11 citations
- Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language ModelsZhong Zhang, Bang Liu, Junming ShaoACL 2023 · 8 citations
- Better Embeddings with Coupled AdamFelix Stollenwerk, Tobias StollenwerkACL 2025 · 4 citations
Builds on1
Related papers
- Weight Tying Biases Token Embeddings Towards the Output SpaceAntonio Lopardo, Avyukth Harish, Catherine Arnett, Akshat GuptaACL Findings 2026
- Representation Projection Invariance Mitigates Representation CollapseAnastasia Razdaibiedina, Ashish Khetan, Zohar S. Karnin, Daniel Khashabi et al.EMNLP Findings 2023 · 1 citation
- Improving Variational Autoencoders with Density Gap-based RegularizationJianfei Zhang, Jun Bai, Chenghua Lin, Yanmeng Wang et al.NeurIPS 2022 · 11 citations
- A Theoretical Analysis of the Repetition Problem in Text GenerationZihao Fu, Wai Lam, Anthony Man-Cho So, Bei ShiAAAI 2021 · 114 citations
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality RegularizationJunlin He, Jinxiao Du, Wei MaNeurIPS 2024 · 19 citations
