Lune

EMNLP Findings2022

Different Tunes Played with Equal Skill: Exploring a Unified Optimization Subspace for Parameter-Efficient Tuning

Jing Yi, Weize Chen, Yujia Qin, Yankai Lin, Ning Ding, Xu Han, Zhiyuan Liu, Maosong Sun, Jie Zhou

Abstract

Delta tuning (DET, also known as parameterefficient tuning) is deemed as the new paradigm for using pre-trained language models (PLMs). Up to now, various DETs with distinct design elements have been proposed, achieving performance on par with fine-tuning. However, the mechanisms behind the above success are still under-explored, especially the connections among various DETs. To fathom the mystery, we hypothesize that the adaptations of different DETs could all be reparameterized as low-dimensional optimizations in a unified optimization subspace, which could be found by jointly decomposing independent solutions of different DETs. Then we explore the connections among different DETs by conducting optimization within the subspace. In experiments, we find that, for a certain DET, conducting optimization simply in the subspace could achieve comparable performance to its original space, and the found solution in the subspace could be transferred to another DET and achieve nontrivial performance. We also visualize the performance landscape of the subspace, and find that, there exists a substantial region where different DETs all perform well. Finally, we extend our analysis and show the strong connections between fine-tuning and DETs. The codes are publicly available at https://github. com/thunlp/Unified-DeltaTuning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
  4. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet ClassificationFrancesco Barbieri, José Camacho-Collados, Luis Espinosa Anke, Leonardo NevesEMNLP Findings 2020 · 720 citations
  5. QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations

Related papers

  1. Sparse Structure Search for Delta TuningShengding Hu, Zhen Zhang, Ning Ding, Yadao Wang et al.NeurIPS 2022 · 27 citations
  2. ADePT: Adaptive Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningPengwei Tang, Xiaolin Hu, Yong LiuICLR 2025
  3. EPT: Efficient Prompt Tuning by Multi-Space Projection and Prompt FusionPengxiang Lan, Enneng Yang, Yuting Liu, Guibing Guo et al.AAAI 2025 · 4 citations
  4. FPT: Improving Prompt Tuning Efficiency via Progressive TrainingYufei Huang, Yujia Qin, Huadong Wang, Yichun Yin et al.EMNLP Findings 2022 · 3 citations
  5. Late Prompt Tuning: A Late Prompt Could Be Better Than Many PromptsXiangyang Liu, Tianxiang Sun, Xuanjing Huang, Xipeng QiuEMNLP Findings 2022 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines