Customizable Combination of Parameter-Efficient Modules for Multi-Task Learning
Haowen Wang, Tao Sun, Congyun Jin, Yingbo Wang, Yibo Fan, Yunqi Xu, Yuliang Du, Cong Fan
Abstract
Modular and composable transfer learning is an emerging direction in the field of Parameter Efficient Fine-Tuning, as it enables neural networks to better organize various aspects of knowledge, leading to improved cross-task generalization. In this paper, we introduce a novel approach Customized Polytropon (C-Poly) that combines task-common skills and task-specific skills, while the skill parameters being highly parameterized using low-rank techniques. Each task is associated with a customizable number of exclusive specialized skills and also benefits from skills shared with peer tasks. A skill assignment matrix is jointly learned. To evaluate our approach, we conducted extensive experiments on the Super-NaturalInstructions and the SuperGLUE benchmarks. Our findings demonstrate that C-Poly outperforms fully-shared, task-specific, and skill-indistinguishable baselines, significantly enhancing the sample efficiency in multi-task learning scenarios.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- Customizing Language Models with Instance-wise LoRA for Sequential Recommendation cites LoRA: Low-Rank Adaptation of Large Language Models
- Customizing Language Models with Instance-wise LoRA for Sequential Recommendation cites Language Models are Few-Shot Learners
- Customizing Language Models with Instance-wise LoRA for Sequential Recommendation cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- Customizing Language Models with Instance-wise LoRA for Sequential Recommendation cites Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
- MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning cites LoRA: Low-Rank Adaptation of Large Language Models
- MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning cites Language Models are Few-Shot Learners
- MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA cites LoRA: Low-Rank Adaptation of Large Language Models
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Language Models are Few-Shot Learners
Cited by top-tier papers3
- Customizing Language Models with Instance-wise LoRA for Sequential RecommendationXiaoyu Kong, Jiancan Wu, An Zhang, Leheng Sheng et al.NeurIPS 2024 · 66 citations
- MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task LearningYufei Ma, Zihan Liang, Huangyu Dai, Ben Chen et al.EMNLP 2024 · 4 citations
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRASangmin Bae, Adam Fisch, Hrayr Harutyunyan, Ziwei Ji et al.ICLR 2025
Builds on11
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
- Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few ExamplesEleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin et al.ICLR 2020 · 692 citations
- Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual ModelsZirui Wang, Yulia Tsvetkov, Orhan Firat, Yuan CaoICLR 2021 · 241 citations
Related papers
- Low-Rank Updates of pre-trained Weights for Multi-Task LearningAlexandre Audibert, Massih-Reza Amini, Konstantin Usevich, Marianne ClauselACL Findings 2023 · 2 citations
- Parameter-efficient Multi-task Fine-tuning for Transformers via Shared HypernetworksRabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, James HendersonACL 2021
- SPoT: Better Frozen Model Adaptation through Soft Prompt TransferTu Vu, Brian Lester, Noah Constant, Rami Al-Rfou' et al.ACL 2022 · 332 citations
- Efficient Multi-task Reinforcement Learning with Cross-Task Policy GuidanceJinmin He, Kai Li, Yifan Zang, Haobo Fu et al.NeurIPS 2024 · 11 citations
- A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language UnderstandingZhe Yang, Yi Huang, Yaqin Chen, Mengfei Guo et al.ACL Findings 2026
