EMNLP Findings2023
Representation Projection Invariance Mitigates Representation Collapse
Anastasia Razdaibiedina, Ashish Khetan, Zohar S. Karnin, Daniel Khashabi, Vivek Madan
Abstract
Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization.
In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). Additionally, REPINA improves out-of-distribution performance. We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- Prompt and Parameter Co-Optimization for Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging cites LoRA: Low-Rank Adaptation of Large Language Models
- LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging cites Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
- MODEL SOUPS NEED ONLY ONE INGREDIENT cites LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
- MODEL SOUPS NEED ONLY ONE INGREDIENT cites Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Cited by top-tier papers4
- Prompt and Parameter Co-Optimization for Large Language ModelsXiaohe Bo, Rui Li, Zexu Sun, Quanyu Dai et al.ICLR 2026 · 2 citations
- LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model MergingKe Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jiménez et al.ICLR 2025
- Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RLTian-Shuo Liu, Chengxing Jia, Haoyu Liu, Pengyuan Wang et al.ICML 2026
- MODEL SOUPS NEED ONLY ONE INGREDIENTAlireza Abdollahpourrostam, Nikolaos Dimitriadis, Adam Hazimeh, Pascal FrossardICML 2026
Builds on21
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Fine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionAnanya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma et al.ICLR 2022 · 911 citations
- A Survey of Data Augmentation Approaches for NLPSteven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar et al.ACL Findings 2021 · 708 citations
- Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningBeliz Gunel, Jingfei Du, Alexis Conneau, Veselin StoyanovICLR 2021 · 595 citations
Related papers
- HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation PerturbationHongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang et al.ACL 2023 · 5 citations
- Domain Adversarial Fine-Tuning as an Effective RegularizerGiorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, Ion AndroutsopoulosEMNLP Findings 2020 · 11 citations
- Better Fine-Tuning by Reducing Representational CollapseArmen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal et al.ICLR 2021 · 20 citations
- Mixout: Effective Regularization to Finetune Large-scale Pretrained Language ModelsCheolhyoung Lee, Kyunghyun Cho, Wanmo KangICLR 2020 · 233 citations
- Revisiting Representation Degeneration Problem in Language ModelingZhong Zhang, Chongming Gao, Cong Xu, Rui Miao et al.EMNLP Findings 2020 · 19 citations
