Lune

ACL2023Top-tier venue

HINT: Hypernetwork Instruction Tuning for Efficient Zero- and Few-Shot Generalisation

Hamish Ivison, Akshita Bhagia, Yizhong Wang, Hannaneh Hajishirzi, Matthew E. Peters

Abstract

Recent NLP models have shown the remarkable ability to effectively generalise 'zero-shot' to new tasks using only natural language instructions as guidance. However, many of these approaches suffer from high computational costs due to their reliance on concatenating lengthy instructions with every input example, resulting in costly reprocessing of the instruction. To avoid this, we introduce Hypernetworks for INstruction Tuning (HINT), which convert task instructions and examples into parameter-efficient modules inserted into an underlying model using a pretrained text encoder, eliminating the need to include instructions in the model input. The hypernetwork in HINT also produces an encoded instruction, which we concatenate with encoded inputs during decoding to further improve performance. HINT models outperform strong state-of-theart baselines by over 10% when controlling for compute (measured in FLOPs). By converting instructions into modules, HINT models can effectively disregard the length of instructions and few-shot example inputs in terms of compute usage. As a result, HINT can enhance its performance by up to 25% by incorporating additional few-shot data, while utilizing only up to 5% more compute. This combines the strengths of parameter-efficient fine-tuning and in-context learning. We release our code publicly 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning cites LoRA: Low-Rank Adaptation of Large Language Models
  • Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning cites Finetuned Language Models are Zero-Shot Learners
  • Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
  • Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning cites Cross-Task Generalization via Natural Language Crowdsourcing Instructions
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation cites LoRA: Low-Rank Adaptation of Large Language Models
  • Language Constrained Multimodal Hyper Adapter For Many-to-Many Multimodal Summarization cites LoRA: Low-Rank Adaptation of Large Language Models
  • HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing cites LoRA: Low-Rank Adaptation of Large Language Models
  • HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing cites Language Constrained Multimodal Hyper Adapter For Many-to-Many Multimodal Summarization
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites LoRA: Low-Rank Adaptation of Large Language Models
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Finetuned Language Models are Zero-Shot Learners
  • Instant Personalized Large Language Model Adaptation via Hypernetwork cites Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
  • Cross-Task Generalization via Natural Language Crowdsourcing Instructions cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Cross-Task Generalization via Natural Language Crowdsourcing Instructions cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction TuningTed Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis et al.ICLR 2024 · 169 citations
  2. Instant Personalized Large Language Model Adaptation via HypernetworkZhaoxuan Tan, Zixuan Zhang, Haoyang Wen, Zheng Li et al.ACL 2026 · 7 citations
  3. HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-PlayingShihao Yang, Zhicong Lu, Yong Yang, Bo Lv et al.AAAI 2026 · 2 citations
  4. LoRAGen: Structure-Aware Weight Space Learning for LoRA GenerationHao Huang, Jingtao Ding, Mengqi Liao, Xin Wang et al.ICLR 2026
  5. Language Constrained Multimodal Hyper Adapter For Many-to-Many Multimodal SummarizationNayu Liu, Fanglong Yao, Haoran Luo, Yong Yang et al.ACL 2025

Builds on15

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  3. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
  4. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
  5. Cross-Task Generalization via Natural Language Crowdsourcing InstructionsSwaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh HajishirziACL 2022 · 887 citations

Related papers

  1. Few-Shot Learning with Siamese Networks and Label TuningThomas Müller, Guillermo Pérez-Torró, Marc Franco-SalvadorACL 2022
  2. PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningJiaru Zou, Mengyu Zhou, Tao Li, Shi Han et al.EMNLP Findings 2024 · 6 citations
  3. HyperPrompt: Prompt-based Task-Conditioning of TransformersYun He, Huaixiu Steven Zheng, Yi Tay, Jai Prakash Gupta et al.ICML 2022 · 110 citations
  4. M²PT: Multimodal Prompt Tuning for Zero-shot Instruction LearningTaowen Wang, Yiyang Liu, James Liang, Junhan Zhao et al.EMNLP 2024 · 31 citations
  5. InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction TuningPrakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri et al.EMNLP 2022 · 26 citations
Dusk over the sea between two cliffs drawn in fine vertical lines