Lune

EMNLP Findings2023

Representation Projection Invariance Mitigates Representation Collapse

Anastasia Razdaibiedina, Ashish Khetan, Zohar S. Karnin, Daniel Khashabi, Vivek Madan

Abstract

Fine-tuning contextualized representations learned by pre-trained language models remains a prevalent practice in NLP. However, fine-tuning can lead to representation degradation (also known as representation collapse), which may result in instability, sub-optimal performance, and weak generalization.

In this paper, we propose Representation Projection Invariance (REPINA), a novel regularization method to maintain information content of representation and reduce representation collapse during fine-tuning by discouraging undesirable changes in the representations. We study the empirical behavior of the proposed regularization in comparison to 5 comparable baselines across 13 language understanding tasks (GLUE benchmark and six additional datasets). When evaluating in-domain performance, REPINA consistently outperforms other baselines on most tasks (10 out of 13). Additionally, REPINA improves out-of-distribution performance. We also demonstrate its effectiveness in few-shot settings and robustness to label perturbation. As a by-product, we extend previous studies of representation collapse and propose several metrics to quantify it. Our empirical findings show that our approach is significantly more effective at mitigating representation collapse. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Prompt and Parameter Co-Optimization for Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging cites LoRA: Low-Rank Adaptation of Large Language Models
  • LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging cites Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
  • MODEL SOUPS NEED ONLY ONE INGREDIENT cites LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging
  • MODEL SOUPS NEED ONLY ONE INGREDIENT cites Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Prompt and Parameter Co-Optimization for Large Language ModelsXiaohe Bo, Rui Li, Zexu Sun, Quanyu Dai et al.ICLR 2026 · 2 citations
  2. LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model MergingKe Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jiménez et al.ICLR 2025
  3. Clipping Low-Probability Tokens in SFT Yields a Generalizable Initialization for RLTian-Shuo Liu, Chengxing Jia, Haoyu Liu, Pengyuan Wang et al.ICML 2026
  4. MODEL SOUPS NEED ONLY ONE INGREDIENTAlireza Abdollahpourrostam, Nikolaos Dimitriadis, Adam Hazimeh, Pascal FrossardICML 2026

Builds on21

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionAnanya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma et al.ICLR 2022 · 911 citations
  4. A Survey of Data Augmentation Approaches for NLPSteven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar et al.ACL Findings 2021 · 708 citations
  5. Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningBeliz Gunel, Jingfei Du, Alexis Conneau, Veselin StoyanovICLR 2021 · 595 citations

Related papers

  1. HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation PerturbationHongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang et al.ACL 2023 · 5 citations
  2. Domain Adversarial Fine-Tuning as an Effective RegularizerGiorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, Ion AndroutsopoulosEMNLP Findings 2020 · 11 citations
  3. Better Fine-Tuning by Reducing Representational CollapseArmen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal et al.ICLR 2021 · 20 citations
  4. Mixout: Effective Regularization to Finetune Large-scale Pretrained Language ModelsCheolhyoung Lee, Kyunghyun Cho, Wanmo KangICLR 2020 · 233 citations
  5. Revisiting Representation Degeneration Problem in Language ModelingZhong Zhang, Chongming Gao, Cong Xu, Rui Miao et al.EMNLP Findings 2020 · 19 citations
Dusk over the sea between two cliffs drawn in fine vertical lines