ACL Findings2023
Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning
Hyeonmin Ha, Soyoung Jung, Jinsol Park, Minjoon Seo, Seung-won Hwang, Byung-Gon Chun
Abstract
Prompting has gained tremendous attention as an efficient method for the adaptation of largescale language models. However, prompts often act against human intuition and report unstable performances, which has motivated methods that automatically find effective prompts. One popular approach is gradient-based search, which iteratively updates a (randomly) initialized prompt towards the optimal one with the guide of gradients. We propose a novel regularization method, CoRe, for gradient-based prompt tuning techniques, which guides a prompt to produce a task context properly. CoRe realizes two regularization effects -context attuning and context filtering -that improve prediction performance in a zero-shot incontext learning setting where a model makes inferences only with the prompt tuned by CoRe, without any demonstration examples for incontext learning. Context attuning guides the context generated by the input and the tuned prompt toward embedding the appropriate context for the task. In our theoretical analysis, regularizing the context extends to improving zeroshot in-context learning performance. Context filtering steers the prompt to select only the task-related context so that context attuning solely focuses on creating and sending the right task context. We evaluate CoRe on natural language understanding datasets and two large language models, GPT2-XL and GPT-J. Our training scheme shows performance improvements up to 11.9% on GPT2-XL, and up to 6.3% on GPT-J in zero-shot settings.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- The Power of Scale for Parameter-Efficient Prompt Tuning cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- An Explanation of In-context Learning as Implicit Bayesian Inference cites The Power of Scale for Parameter-Efficient Prompt Tuning
- An Explanation of In-context Learning as Implicit Bayesian Inference cites Language Models are Few-Shot Learners
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
- Language Models are Few-Shot Learners cites UnifiedQA: Crossing Format Boundaries With a Single QA System
Builds on10
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
- An Explanation of In-context Learning as Implicit Bayesian InferenceSang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu MaICLR 2022 · 1,030 citations
- UnifiedQA: Crossing Format Boundaries With a Single QA SystemDaniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal et al.EMNLP Findings 2020 · 514 citations
- The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
Related papers
- Universal Self-Adaptive PromptingXingchen Wan, Ruoxi Sun, Hootan Nakhost, Hanjun Dai et al.EMNLP 2023 · 4 citations
- Context Tuning for In-Context OptimizationJack Lu, Ryan Teehan, Zhenbang Yang, Mengye RenICML 2026
- Consistency-guided Prompt Learning for Vision-Language ModelsShuvendu Roy, Ali EtemadICLR 2024 · 102 citations
- On the Role of Attention in Prompt-tuningSamet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, Christos ThrampoulidisICML 2023 · 67 citations
- Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman et al.EMNLP Findings 2023 · 13 citations
