Lune

EMNLP Findings2022

SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters

Shwai He, Liang Ding, Daize Dong, Jeremy Zhang, Dacheng Tao

Abstract

Adapter Tuning, which freezes the pretrained language models (PLMs) and only fine-tunes a few extra modules, has become an appealing efficient alternative to the full model finetuning. Although computationally efficient, the recent adapters often increase parameters (e.g. bottleneck dimension) for matching the performance of full model fine-tuning, which we argue goes against their original intention. In this work, we re-examine the parameterefficiency of adapters through the lens of network pruning (we name such plug-in concept as SparseAdapter) and find that SparseAdapter can achieve comparable or better performance than standard adapters when the sparse ratio reaches up to 80%. Based on our findings, we introduce an easy but effective setting "Large-Sparse" to improve the model capacity of adapters under the same parameter budget. Experiments on five competitive adapters upon three advanced PLMs show that with proper sparse method (e.g. SNIP) and ratio (e.g. 40%) SparseAdapter can consistently outperform their corresponding counterpart. Encouragingly, with the Large-Sparse setting, we can obtain further appealing gains, even outperforming the full fine-tuning by a large margin. Our code will be released at: https: //github.com/Shwai-He/SparseAdapter.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

25 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion Recognition cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • ReFT: Representation Finetuning for Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • ReFT: Representation Finetuning for Language Models cites Towards a Unified View of Parameter-Efficient Transfer Learning
  • Towards a Unified View of Parameter-Efficient Transfer Learning cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation cites LoRA: Low-Rank Adaptation of Large Language Models
  • QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation cites ReFT: Representation Finetuning for Language Models
  • QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation cites Towards a Unified View of Parameter-Efficient Transfer Learning
  • Reinforcement Learning-based Recommender Systems with Large Language Models for State Reward and Action Modeling cites LoRA: Low-Rank Adaptation of Large Language Models
  • Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy cites LoRA: Low-Rank Adaptation of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers25

  1. ReFT: Representation Finetuning for Language ModelsZhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger et al.NeurIPS 2024 · 233 citations
  2. Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision RedundancyShibo Jie, Haoqing Wang, Zhi-Hong DengICCV 2023 · 50 citations
  3. QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor AdaptationZhuo Chen, Rumen Dangovski, Charlotte Loh, Owen Dugan et al.NeurIPS 2024 · 38 citations
  4. Reinforcement Learning-based Recommender Systems with Large Language Models for State Reward and Action ModelingJie Wang, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. JoseSIGIR 2024 · 27 citations
  5. MSE-Adapter: A Lightweight Plugin Endowing LLMs with the Capability to Perform Multimodal Sentiment Analysis and Emotion RecognitionYang Yang, Xunde Dong, Yupeng QiangAAAI 2025 · 19 citations

Builds on11

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
  4. Picking Winning Tickets Before Training by Preserving Gradient FlowChaoqi Wang, Guodong Zhang, Roger B. GrosseICLR 2020 · 743 citations
  5. Rigging the Lottery: Making All Tickets WinnersUtku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro et al.ICML 2020 · 723 citations

Related papers

  1. APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and InferenceBowen Zhao, Hannaneh Hajishirzi, Qingqing CaoICML 2024 · 31 citations
  2. On the Effectiveness of Adapter-based Tuning for Pretrained Language Model AdaptationRuidan He, Linlin Liu, Hai Ye, Qingyu Tan et al.ACL 2021
  3. Sparsity May Be All You Need: Sparse Random Parameter AdaptationJesus Rios, Pierre L. Dognin, Ronny Luss, Karthikeyan Natesan RamamurthyEMNLP Findings 2025
  4. AdaMix: Mixture-of-Adaptations for Parameter-efficient Model TuningYaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu et al.EMNLP 2022 · 65 citations
  5. MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-TuningPengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang et al.ACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines