Lune

ACL Findings2023

Low-Rank Updates of pre-trained Weights for Multi-Task Learning

Alexandre Audibert, Massih-Reza Amini, Konstantin Usevich, Marianne Clausel

Abstract

Multi-Task Learning used with pre-trained models has been quite popular in the field of Natural Language Processing in recent years. This framework remains still challenging due to the complexity of the tasks and the challenges associated with fine-tuning large pre-trained models. In this paper, we propose a new approach for Multi-task learning which is based on stacking the weights of Neural Networks as a tensor. We show that low-rank updates in the canonical polyadic tensor decomposition of this tensor of weights lead to a simple, yet efficient algorithm, which without loss of performance allows to reduce considerably the model parameters. We investigate the interactions between tasks inside the model as well as the inclusion of sparsity to find the best tensor rank and to increase the compression rate. Our strategy is consistent with recent efforts that attempt to use constraints to fine-tune some model components. More precisely, we achieve equivalent performance as the state-of-the-art on the General Language Understanding Evaluation benchmark by training only 0.3% of the parameters per task while not modifying the baseline weights.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining cites LoRA: Low-Rank Adaptation of Large Language Models
  • SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining cites Parameter-Efficient Transfer Learning with Diff Pruning
  • Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data cites Understanding and Improving Information Transfer in Multi-Task Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. SLTrain: a sparse plus low rank approach for parameter and memory efficient pretrainingAndi Han, Jiaxiang Li, Wei Huang, Mingyi Hong et al.NeurIPS 2024 · 54 citations
  2. Efficient Pareto Manifold Learning with Low-Rank StructureWeiyu Chen, James T. KwokICML 2024 · 11 citations
  3. CALM: Consensus-Aware Localized Merging for Multi-Task LearningKunda Yan, Min Zhang, Sen Cui, Zikun Qu et al.ICML 2025

Builds on6

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Understanding and Improving Information Transfer in Multi-Task LearningSen Wu, Hongyang R. Zhang, Christopher RéICLR 2020 · 183 citations
  3. Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less DataJonathan Pilault, Amine Elhattami, Christopher J. PalICLR 2021 · 105 citations
  4. Exploring extreme parameter compression for pre-trained language modelsBenyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang et al.ICLR 2022 · 23 citations
  5. Parameter-Efficient Transfer Learning with Diff PruningDemi Guo, Alexander M. Rush, Yoon KimACL 2021

Related papers

  1. Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuningChongjie Si, Xuehui Wang, Xue Yang, Zhengqin Xu et al.ICLR 2025
  2. Multi-Task Learning via Generalized Tensor Trace NormYi Zhang, Yu Zhang, Wei WangKDD 2021 · 8 citations
  3. Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product OperatorsPeiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Zhi-Yuan Xie et al.ACL 2021
  4. Tensorized Embedding LayersOleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhabova, Elena Orlova et al.EMNLP Findings 2020 · 30 citations
  5. Saten: Sparse Augmented Tensor Networks for Post-Training Compression of Large Language ModelsRyan Solgi, Kai Zhen, Rupak Vignesh Swaminathan, Nathan Susanj et al.EMNLP Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines