Lune

ACL Findings2023

Residual Prompt Tuning: improving prompt tuning with residual reparameterization

Anastasia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, Amjad Almahairi

Abstract

Prompt tuning is one of the successful approaches for parameter-efficient tuning of pretrained language models. Despite being arguably the most parameter-efficient (tuned soft prompts constitute < 0.1% of total parameters), it typically performs worse than other efficient tuning methods and is quite sensitive to hyper-parameters. In this work, we introduce RESIDUAL PROMPT TUNING -a simple and efficient method that significantly improves the performance and stability of prompt tuning. We propose to reparameterize soft prompt embeddings using a shallow network with a residual connection. Our experiments show that RESIDUAL PROMPT TUNING significantly outperforms prompt tuning on SuperGLUE benchmark across T5-Large, T5-Base and BERT-Base models. Notably, our method reaches +7 points improvement over prompt tuning with T5-Base and allows to reduce the prompt length by ×10 without hurting performance. In addition, we show that our approach is robust to the choice of learning rate and prompt initialization, and is effective in few-shot settings. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

26 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer cites LoRA: Low-Rank Adaptation of Large Language Models
  • SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
  • SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer cites Language Models are Few-Shot Learners
  • SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer cites Calibrate Before Use: Improving Few-shot Performance of Language Models
  • Compacter: Efficient Low-Rank Hypercomplex Adapter Layers cites Language Models are Few-Shot Learners
  • Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation cites LoRA: Low-Rank Adaptation of Large Language Models
  • Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptation cites LoRA-GA: Low-Rank Adaptation with Gradient Approximation
  • All You Need is One: Capsule Prompt Tuning with a Single Vector cites LoRA: Low-Rank Adaptation of Large Language Models
  • All You Need is One: Capsule Prompt Tuning with a Single Vector cites Language Models are Few-Shot Learners
  • A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space cites LoRA: Low-Rank Adaptation of Large Language Models
  • A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
  • LoRA-GA: Low-Rank Adaptation with Gradient Approximation cites LoRA: Low-Rank Adaptation of Large Language Models
  • Calibrate Before Use: Improving Few-shot Performance of Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers26

  1. DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
  2. LoRA-GA: Low-Rank Adaptation with Gradient ApproximationShaowen Wang, Linxi Yu, Jian LiNeurIPS 2024 · 194 citations
  3. All You Need is One: Capsule Prompt Tuning with a Single VectorYiyang Liu, James Liang, Heng Fan, Wenhao Yang et al.NeurIPS 2025 · 16 citations
  4. A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-SpaceBingjie Zhang, Yibo Yang, Renzhe, Dandan Guo et al.ICLR 2026 · 12 citations
  5. Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank AdaptationShiwei Li, Xiandi Luo, Haozhao Wang, Xing Tang et al.NeurIPS 2025 · 10 citations

Builds on7

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
  4. Compacter: Efficient Low-Rank Hypercomplex Adapter LayersRabeeh Karimi Mahabadi, James Henderson, Sebastian RuderNeurIPS 2021 · 700 citations
  5. SPoT: Better Frozen Model Adaptation through Soft Prompt TransferTu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' et al.ACL 2022 · 332 citations

Related papers

  1. APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language ModelsQifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu et al.EMNLP 2023 · 25 citations
  2. Decomposed Prompt Tuning via Low-Rank ReparameterizationYao Xiao, Lu Xu, Jiaxi Li, Wei Lu et al.EMNLP Findings 2023 · 4 citations
  3. Late Prompt Tuning: A Late Prompt Could Be Better Than Many PromptsXiangyang Liu, Tianxiang Sun, Xuanjing Huang, Xipeng QiuEMNLP Findings 2022 · 15 citations
  4. FPT: Improving Prompt Tuning Efficiency via Progressive TrainingYufei Huang, Yujia Qin, Huadong Wang, Yichun Yin et al.EMNLP Findings 2022 · 3 citations
  5. EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt FusionPengxiang Lan, Enneng Yang, Yuting Liu, Guibing Guo et al.AAAI 2025 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines