Lune

ACL Findings2023

Robustness of Learning from Task Instructions

Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, Wenpeng Yin

Abstract

Traditional supervised learning mostly works on individual tasks and requires training on a large set of task-specific examples. This paradigm seriously hinders the development of task generalization since preparing a taskspecific example set is costly. To build a system that can quickly and easily generalize to new tasks, textual instructions have been adopted as an emerging trend of supervision recently. These instructions give the model the definition of the task and allow the model to output the appropriate answer based on the instructions and inputs. However, task instructions are often expressed in different forms, which can be interpreted from two threads (Lou et al., 2023): firstly, some instructions are short sentences and are pretrained language model (PLM) oriented, such as prompts, while other instructions are paragraphs and are human-oriented, such as those annotation guidelines in Amazon MTurk; secondly, different end-users very likely explain the same task with instructions of different textual expressions. A robust system for task generalization should be able to handle any new tasks regardless of the variability of instructions. However, the system robustness in dealing with a wide range of instructions is still unexplored. This work investigates the system robustness when the instructions of new tasks are (i) manipulated, (ii) paraphrased, or (iii) from different levels of conciseness. To our knowledge, this is the first work that systematically studies how robust a PLM is when it is supervised by instructions with different factors of variability. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

14 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Understanding the Prompt Sensitivity cites Evaluating the Zero-shot Robustness of Instruction-tuned Language Models
  • Understanding the Prompt Sensitivity cites Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
  • Understanding the Prompt Sensitivity cites Calibrate Before Use: Improving Few-shot Performance of Language Models
  • Evaluating the Zero-shot Robustness of Instruction-tuned Language Models cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Evaluating the Zero-shot Robustness of Instruction-tuned Language Models cites Cross-Task Generalization via Natural Language Crowdsourcing Instructions
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Calibrate Before Use: Improving Few-shot Performance of Language Models
  • On the Worst Prompt Performance of Large Language Models cites Evaluating the Zero-shot Robustness of Instruction-tuned Language Models
  • MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction Following cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction Following cites Cross-Task Generalization via Natural Language Crowdsourcing Instructions
  • Cross-Task Generalization via Natural Language Crowdsourcing Instructions cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Cross-Task Generalization via Natural Language Crowdsourcing Instructions cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Cross-Task Generalization via Natural Language Crowdsourcing Instructions cites Calibrate Before Use: Improving Few-shot Performance of Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers14

  1. Understanding the Prompt SensitivityYang Liu, Chenhui ChuACL 2026 · 192 citations
  2. Evaluating the Zero-shot Robustness of Instruction-tuned Language ModelsJiuding Sun, Chantal Shaib, Byron C. WallaceICLR 2024 · 75 citations
  3. On the Worst Prompt Performance of Large Language ModelsBowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou et al.NeurIPS 2024 · 65 citations
  4. BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled OptimizationXueyang Zhou, Guiyao Tie, Guowen Zhang, Hechang Wang et al.NeurIPS 2025 · 50 citations
  5. MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction FollowingRenze Lou, Kai Zhang, Jian Xie, Yuxuan Sun et al.ICLR 2024 · 39 citations

Builds on9

  1. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
  2. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. Cross-Task Generalization via Natural Language Crowdsourcing InstructionsSwaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh HajishirziACL 2022 · 887 citations
  5. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe et al.EMNLP 2022 · 634 citations

Related papers

  1. Contrastive Instruction TuningTianyi Yan, Fei Wang, James Y. Huang, Wenxuan Zhou et al.ACL Findings 2024 · 3 citations
  2. Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksYizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi et al.EMNLP 2022 · 238 citations
  3. What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic EnvironmentMatthew Finlayson, Kyle Richardson, Ashish Sabharwal, Peter ClarkEMNLP 2022 · 8 citations
  4. Controlled Text Generation with Natural Language InstructionsWangchunshu Zhou, Yuchen Eleanor Jiang, Ethan Wilcox, Ryan Cotterell et al.ICML 2023 · 121 citations
  5. Pre-Training to Learn in ContextYuxian Gu, Li Dong, Furu Wei, Minlie HuangACL 2023 · 18 citations
Dusk over the sea between two cliffs drawn in fine vertical lines