Towards a Unified View of Parameter-Efficient Transfer Learning
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig
Abstract
Fine-tuning large pretrained language models on downstream tasks has become the de-facto learning paradigm in NLP. However, conventional approaches finetune all the parameters of the pretrained model, which becomes prohibitive as the model size and the number of tasks grow. Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance. While effective, the critical ingredients for success and the connections among the various methods are poorly understood. In this paper, we break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them. Specifically, we re-frame them as modifications to specific hidden states in pretrained models, and define a set of design dimensions along which different methods vary, such as the function to compute the modification and the position to apply the modification. Through comprehensive empirical studies across machine translation, text summarization, language understanding, and text classification benchmarks, we utilize the unified view to identify important design choices in previous methods. Furthermore, our unified framework enables the transfer of design elements across different approaches, and as a result we are able to instantiate new parameter-efficient fine-tuning methods that tune less parameters than previous methods while being more effective, achieving comparable results to fine-tuning all parameters on all four tasks. 1 Order determined by random dice rolling.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
344 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers cites The Power of Scale for Parameter-Efficient Prompt Tuning
- Compacter: Efficient Low-Rank Hypercomplex Adapter Layers cites Language Models are Few-Shot Learners
- The Power of Scale for Parameter-Efficient Prompt Tuning cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites The Power of Scale for Parameter-Efficient Prompt Tuning
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Language Models are Few-Shot Learners
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction cites Language Models are Few-Shot Learners
- AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition cites The Power of Scale for Parameter-Efficient Prompt Tuning
- AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition cites Language Models are Few-Shot Learners
- DoRA: Weight-Decomposed Low-Rank Adaptation cites The Power of Scale for Parameter-Efficient Prompt Tuning
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites The Power of Scale for Parameter-Efficient Prompt Tuning
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Language Models are Few-Shot Learners
- Extracting Training Data from Large Language Models cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Extracting Training Data from Large Language Models cites Language Models are Few-Shot Learners
Cited by top-tier papers344
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
- AdaptFormer: Adapting Vision Transformers for Scalable Visual RecognitionShoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang et al.NeurIPS 2022 · 1,291 citations
- The Flan Collection: Designing Data and Methods for Effective Instruction TuningShayne Longpre, Le Hou, Tu Vu, Albert Webson et al.ICML 2023 · 908 citations
- DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
- KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation ExtractionXiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng et al.WWW 2022 · 488 citations
Builds on8
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Compacter: Efficient Low-Rank Hypercomplex Adapter LayersRabeeh Karimi Mahabadi, James Henderson, Sebastian RuderNeurIPS 2021 · 700 citations
- The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
Related papers
- HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language TasksZhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang et al.ACL Findings 2023 · 12 citations
- Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningRunxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan et al.EMNLP 2021 · 129 citations
- Inducer-tuning: Connecting Prefix-tuning and Adapter-tuningYifan Chen, Devamanyu Hazarika, Mahdi Namazifar, Yang Liu et al.EMNLP 2022 · 4 citations
- SPoT: Better Frozen Model Adaptation through Soft Prompt TransferTu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' et al.ACL 2022 · 332 citations
- Prototype-based HyperAdapter for Sample-Efficient Multi-task TuningHao Zhao, Jie Fu, Zhaofeng HeEMNLP 2023 · 3 citations
