EMNLP Findings2023
MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension
Guoxin Chen, Yiming Qian, Bowen Wang, Liangzhi Li
摘要
The large language models have achieved superior performance on various natural language tasks. One major drawback of such approaches is they are resource-intensive in fine-tuning new datasets. Soft-prompt tuning presents a resource-efficient solution to fine-tune the pretrained language models (PLMs) while keeping their weight frozen. Existing soft prompt methods mainly focus on designing the inputindependent prompts that steer the model to fit the domain of the new dataset. Those methods often ignore the fine-grained information about the task and context of the text. In this paper, we propose a multi-level prompt tuning (MPrompt) method for machine reading comprehension. It utilizes prompts at task-specific, domain-specific, and context-specific levels to enhance the comprehension of input semantics at different granularities. We also propose an independence constraint to steer each domainspecific prompt to focus on information within its domain to avoid redundancy. Moreover, we present a prompt generator that incorporates context-related knowledge in the prompt generation to enhance contextual relevancy. We conducted extensive experiments on 12 benchmarks of various QA formats and achieved an average improvement of 1.94% over the stateof-the-art methods 1 .
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts 引用了 Language Models are Few-Shot Learners
- Language Models are Few-Shot Learners 引用了 Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks 引用了 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
引用它的顶会 Paper1
它引用的 Paper11
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni 等NeurIPS 2020 · 19,162 次引用
- Do Vision Transformers See Like Convolutional Neural Networks?Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang 等NeurIPS 2021 · 1,553 次引用
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad 等ACL 2020 · 1,224 次引用
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace 等EMNLP 2020 · 1,162 次引用
相关 Paper
- APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language ModelsQifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu 等EMNLP 2023 · 25 次引用
- Multitask Pre-training of Modular Prompt for Chinese Few-Shot LearningTianxiang Sun, Zhengfu He, Qin Zhu, Xipeng Qiu 等ACL 2023 · 15 次引用
- Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise PromptsMuling Wu, Wenhao Liu, Jianhan Xu, Changze Lv 等EMNLP Findings 2023 · 2 次引用
- PPT: Pre-trained Prompt Tuning for Few-shot LearningYuxian Gu, Xu Han, Zhiyuan Liu, Minlie HuangACL 2022
- Two-stage LLM Fine-tuning with Less Specialization and More GeneralizationYihan Wang, Si Si, Daliang Li, Michal Lukasik 等ICLR 2024 · 45 次引用
