EMNLP Findings2024
HyperLoRA: Efficient Cross-task Generalization via Constrained Low-Rank Adapters Generation
Chuancheng Lv, Lei Li, Shitou Zhang, Gang Chen, Fanchao Qi, Ningyu Zhang, Hai-Tao Zheng
Abstract
Adapting pre-trained language models (PLMs) for cross-task generalization is a crucial research area within the field of NLP. While fine-tuning and in-context learning are effective approaches for adapting LMs to emerging tasks, they can be costly and inefficient. Recently, some researchers have focused on achieving efficient task adaptation via hypernetwork, which is a meta network that generates task-specific weights based on task-oriented information without any optimization. However, the training of hypernetworks often lacks stability since the optimization signal is not straightforward, and the task information is not adequately representative. Moreover, previous works train hypenetworks with the general corpus, which is struggling with few-shot adaptation. To address these issues, we introduce HyperLoRA, a hypernetwork for LoRA parameters generation involving hypernetwork pretraining on instruction-following data and generalization fine-tuning on sparse task data. Furthermore, we utilize a constrained training loss and a gradient-based demonstration selection strategy to enhance the training stability and performance. Experimental results and analysis across four benchmark datasets (P3, S-NI, BBH, and SuperGLUE) demonstrate the proposed approach has flexible generalization ability and superior performance.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- Optimization Inspired Few-Shot Adaptation for Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Optimization Inspired Few-Shot Adaptation for Large Language Models cites Language Models are Few-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation cites LoRA: Low-Rank Adaptation of Large Language Models
- Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
- HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing cites LoRA: Low-Rank Adaptation of Large Language Models
- Training language models to follow instructions with human feedback cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers4
- Instant Personalized Large Language Model Adaptation via HypernetworkZhaoxuan Tan, Zixuan Zhang, Haoyang Wen, Zheng Li et al.ACL 2026 · 7 citations
- Optimization Inspired Few-Shot Adaptation for Large Language ModelsBoyan Gao, Xin Wang, Yibo Yang, David A. CliftonNeurIPS 2025 · 3 citations
- HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-PlayingShihao Yang, Zhicong Lu, Yong Yang, Bo Lv et al.AAAI 2026 · 2 citations
- LoRAGen: Structure-Aware Weight Space Learning for LoRA GenerationHao Huang, Jingtao Ding, Mengqi Liao, Xin Wang et al.ICLR 2026
Builds on24
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
Related papers
- Text-to-LoRA: Instant Transformer AdaptionRujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko LangeICML 2025
- HyperTuning: Toward Adapting Large Language Models without Back-propagationJason Phang, Yi Mao, Pengcheng He, Weizhu ChenICML 2023 · 43 citations
- MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-TuningPengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang et al.ACL 2024
- SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single PassYewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelberg et al.ICML 2026 · 11 citations
- RoseLoRA: Row and Column-wise Sparse Low-rank Adaptation of Pre-trained Language Model for Knowledge Editing and Fine-tuningHaoyu Wang, Tianci Liu, Ruirui Li, Monica Xiao Cheng et al.EMNLP 2024 · 6 citations
