Lune

EMNLP Findings2023

Decomposed Prompt Tuning via Low-Rank Reparameterization

Yao Xiao, Lu Xu, Jiaxi Li, Wei Lu, Xiaoli Li

Abstract

While prompt tuning approaches have achieved competitive performance with high efficiency, we observe that they invariably employ the same initialization process, wherein the soft prompt is either randomly initialized or derived from an existing embedding vocabulary. In contrast to these conventional methods, this study aims to investigate an alternative way to derive soft prompt. Our empirical studies show that the soft prompt typically exhibits a low "intrinsic rank" characteristic. With such observations, we propose decomposed prompt tuning, a novel approach that utilizes low-rank matrices to initialize the soft prompt. Through the low-rank reparameterization, our method significantly reduces the number of trainable parameters while maintaining effectiveness. Experimental results on the SuperGLUE benchmark in both high-resource and low-resource scenarios demonstrate the effectiveness of the proposed method. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
  • TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal cites LoRA: Low-Rank Adaptation of Large Language Models
  • ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
  • ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Calibrate Before Use: Improving Few-shot Performance of Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt TuningZongqian Li, Yixuan Su, Nigel CollierNeurIPS 2025 · 7 citations
  2. EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt FusionPengxiang Lan, Enneng Yang, Yuting Liu, Guibing Guo et al.AAAI 2025 · 4 citations
  3. TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather RemovalHanting Wang, Shengpeng Ji, Shulei Wang, Hai Huang et al.ACM MM 2025
  4. ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningPengwei Tang, Xiaolin Hu, Yong LiuICLR 2025

Builds on20

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
  3. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
  4. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  5. Compacter: Efficient Low-Rank Hypercomplex Adapter LayersRabeeh Karimi Mahabadi, James Henderson, Sebastian RuderNeurIPS 2021 · 700 citations

Related papers

  1. Residual Prompt Tuning: improving prompt tuning with residual reparameterizationAnastasia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis et al.ACL Findings 2023 · 25 citations
  2. DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningZhengxiang Shi, Aldo LipaniICLR 2024 · 46 citations
  3. APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language ModelsQifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu et al.EMNLP 2023 · 25 citations
  4. Multitask Prompt Tuning Enables Parameter-Efficient Transfer LearningZhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris et al.ICLR 2023 · 30 citations
  5. SPoT: Better Frozen Model Adaptation through Soft Prompt TransferTu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' et al.ACL 2022 · 332 citations
Dusk over the sea between two cliffs drawn in fine vertical lines