Lune

ACL Findings2024

LoRA Meets Dropout under a Unified Framework

Sheng Wang, Liheng Chen, Jiyue Jiang, Boyang Xue, Lingpeng Kong, Chuan Wu

Abstract

With the remarkable capabilities, large language models (LLMs) have emerged as essential elements in numerous NLP applications, while parameter-efficient finetuning, especially LoRA, has gained popularity as a lightweight approach for model customization. Meanwhile, various dropout methods, initially designed for full finetuning with all the parameters updated, alleviates overfitting associated with excessive parameter redundancy. Hence, a possible contradiction arises from negligible trainable parameters of LoRA and the effectiveness of previous dropout methods, which has been largely overlooked. To fill this gap, we first confirm that parameter-efficient LoRA is also overfitting-prone. We then revisit transformerspecific dropout methods, and establish their equivalence and distinctions mathematically and empirically. Building upon this comparative analysis, we introduce a unified framework for a comprehensive investigation, which instantiates these methods based on dropping position, structural pattern and compensation measure. Through this framework, we reveal the new preferences and performance comparisons of them when involved with limited trainable parameters. This framework also allows us to amalgamate the most favorable aspects into a novel dropout method named HiddenKey. Extensive experiments verify the remarkable superiority and sufficiency of HiddenKey across multiple models and tasks, which highlights it as the preferred approach for high-performance and parameter-efficient finetuning of LLMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Prefix-Tuning: Optimizing Continuous Prompts for Generation
  • LoRA: Low-Rank Adaptation of Large Language Models cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • Prefix-Tuning: Optimizing Continuous Prompts for Generation cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • The Power of Scale for Parameter-Efficient Prompt Tuning cites Prefix-Tuning: Optimizing Continuous Prompts for Generation
  • Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation cites Prefix-Tuning: Optimizing Continuous Prompts for Generation
  • Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. A Stronger Mixture of Low-Rank Experts for Fine-Tuning Foundation ModelsMengyang Sun, Yihao Wang, Tao Feng, Dan Zhang et al.ICML 2025
  2. Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank AdaptationZhan Zhuang, Xiequn Wang, Wei Li, Yulong Zhang et al.ICML 2025
  3. LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language ModelsJian Liang, Wenke Huang, Guancheng Wan, Qu Yang et al.CVPR 2025
  4. MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of ShardsSheng Wang, Liheng Chen, Pengan Chen, Jingwei Dong et al.ICLR 2025
  5. Does Low Rank Adaptation Lead to Lower Robustness against Training-Time Attacks?Zi Liang, Haibo Hu, Qingqing Ye, Yaxin Xiao et al.ICML 2025

Builds on6

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. R-Drop: Regularized Dropout for Neural NetworksXiaobo Liang, Lijun Wu, Juntao Li, Yue Wang et al.NeurIPS 2021 · 610 citations
  3. The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
  4. Prefix-Tuning: Optimizing Continuous Prompts for GenerationXiang Lisa Li, Percy LiangACL 2021
  5. HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better GeneralizabilityJiaao Chen, Dinghan Shen, Weizhu Chen, Diyi YangACL 2021

Related papers

  1. Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningJing Xu, Jingzhao ZhangICML 2024 · 15 citations
  2. Sparsity May Be All You Need: Sparse Random Parameter AdaptationJesus Rios, Pierre L. Dognin, Ronny Luss, Karthikeyan Natesan RamamurthyEMNLP Findings 2025
  3. MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-TuningPengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang et al.ACL 2024
  4. Uni-LoRA: One Vector is All You NeedKaiyang Li, Shaobo Han, Qing Su, Wei Li et al.NeurIPS 2025 · 10 citations
  5. WeightLoRA: Keep Only Necessary AdaptersAndrey Veprikov, Vladimir Solodkin, Alexander Zyl, Andrey V. Savchenko et al.ACL 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines