ACL Findings2023
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors
Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi
Abstract
Obtaining labeled data to train a model for a task of interest is often expensive. Prior work shows training models on multitask data augmented with task descriptions (prompts) effectively transfers knowledge to new tasks. Towards efficiently building task-specific models, we assume access to a small number (32-1000) of unlabeled target-task examples and use those to retrieve the most similar labeled examples from a large pool of multitask data augmented with prompts. Compared to the current practice of finetuning models on uniformly sampled prompted multitask data (e.g., FLAN, T0), our approach of finetuning on cross-task nearest neighbors is significantly more data-efficient. Using only 2% of the data from the P3 pool without any labeled target-task data, our models outperform strong baselines trained on all available data by 3-30% on 12 out of 14 datasets representing held-out tasks including legal and scientific document QA. Similarly, models trained on cross-task nearest neighbors from SuperNat-uralInstructions, representing about 5% of the pool, obtain comparable performance to stateof-the-art models on 12 held-out tasks from that pool. Moreover, the models produced by our approach also provide a better initialization than single multitask finetuned models for fewshot finetuning on target-task data, as shown by a 2-23% relative improvement over fewshot finetuned T0-3B models on 8 datasets. We publicly release our code. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- In-Context Pretraining: Language Modeling Beyond Document Boundaries cites Retrieval Augmented Language Model Pre-Training
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts cites Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding cites LoRA: Low-Rank Adaptation of Large Language Models
- Finetuned Language Models are Zero-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Cited by top-tier papers16
- Reinforcement Learning for Reasoning in Large Language Models with One Training ExampleYiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren et al.NeurIPS 2025 · 314 citations
- Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective RolloutsHaizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura et al.NeurIPS 2025 · 125 citations
- In-Context Pretraining: Language Modeling Beyond Document BoundariesWeijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou et al.ICLR 2024 · 87 citations
- SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence UnderstandingTianyu Yu, Chengyue Jiang, Chao Lou, Shen Huang et al.AAAI 2024 · 30 citations
- Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary DataAlon Albalak, Colin A. Raffel, William Yang WangNeurIPS 2023 · 17 citations
Builds on19
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
Related papers
- Unsupervised Cross-Task Generalization via Retrieval AugmentationBill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian et al.NeurIPS 2022 · 53 citations
- Prompt-free and Efficient Few-shot Learning with Language ModelsRabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias et al.ACL 2022 · 76 citations
- Multitask Prompt Tuning Enables Parameter-Efficient Transfer LearningZhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Feris et al.ICLR 2023 · 30 citations
- Data-Efficient Hate Speech Detection via Cross-Lingual Nearest Neighbor Retrieval with Limited Labeled DataFaeze Ghorbanpour, Daryna Dementieva, Alexander FraserEMNLP 2025
- Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsRobert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni et al.ACL Findings 2022 · 133 citations
