Lune

EMNLP Findings2022

Task Compass: Scaling Multi-task Pre-training with Task Prefix

Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu, Michael Zeng

Abstract

Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models. Existing studies show that multitask learning with large-scale supervised tasks suffers from negative effects across tasks. To tackle the challenge, we propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. We conduct extensive experiments on 40 datasets, which show that our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships. The task relationships reflected by the prefixes align transfer learning performance between tasks. They also suggest directions for data augmentation with complementary tasks, which help our model achieve human-parity results on commonsense reasoning leaderboards. Code

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Finetuned Language Models are Zero-Shot Learners cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Finetuned Language Models are Zero-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Is ChatGPT a General-Purpose Natural Language Processing Task Solver? cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Is ChatGPT a General-Purpose Natural Language Processing Task Solver?Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen et al.EMNLP 2023 · 449 citations
  2. Improved Active Multi-Task Representation Learning via LassoYiping Wang, Yifang Chen, Kevin Jamieson, Simon Shaolei DuICML 2023 · 13 citations
  3. Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel TasksAnwoy Chatterjee, Eshaan Tanwar, Subhabrata Dutta, Tanmoy ChakrabortyACL 2024 · 5 citations
  4. SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by SimulationMatthias Lindemann, Alexander Koller, Ivan TitovACL 2024 · 2 citations

Builds on29

  1. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  2. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  3. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  4. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  5. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations

Related papers

  1. UniEvent: Unified Generative Model with Multi-Dimensional Prefix for Zero-Shot Event-Relational ReasoningZhengwei Tao, Zhi Jin, Haiyan Zhao, Chengfeng Dou et al.ACL 2023 · 2 citations
  2. Commonsense Knowledge Transfer for Pre-trained Language ModelsWangchunshu Zhou, Ronan Le Bras, Yejin ChoiACL Findings 2023 · 3 citations
  3. Muppet: Massive Multi-task Representations with Pre-FinetuningArmen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen et al.EMNLP 2021 · 176 citations
  4. Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language InferenceCanming Huang, Weinan He, Yongmei LiuEMNLP Findings 2021 · 8 citations
  5. Visually-Augmented Language ModelingWeizhi Wang, Li Dong, Hao Cheng, Haoyu Song et al.ICLR 2023 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines