ACL Findings2023
Fixed Input Parameterization for Efficient Prompting
Eunbi Choi, Yongrae Jo, Joel Jang, Joonwon Jang, Minjoon Seo
Abstract
Recent works have shown that attaching prompts to the input is effective at conditioning Language Models (LM) to perform specific tasks. However, prompts are always included in the input text during inference, even when they are fixed, thus incurring substantial computational and memory overhead. Also, there is currently no straightforward method of utilizing prompts that are longer than the maximum input length of the LMs without incurring additional costs during inference. We formally define Fixed Input Parameterization (FIP) problem that focuses on injecting the fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input. We show that in scenarios with long fixed prompts, FIP can be up to 280 times more efficient in terms of total FLOPs than previous approaches. We further explore methodologies for FIP and show promising results in persona-dependent conversation, semantic parsing, and zero-shot learning with task instructions. Through these explorations, we show that FIP can be a promising direction for conditioning language models, in scenarios with long and fixed prompts 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
- Doc-to-LoRA: Learning to Instantly Internalize Contexts cites Language Models are Few-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
Cited by top-tier papers1
Builds on12
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 citations
- Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
Related papers
- PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningJiaru Zou, Mengyu Zhou, Tao Li, Shi Han et al.EMNLP Findings 2024 · 6 citations
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction FollowingSeonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun et al.AAAI 2024 · 50 citations
- Memory-Space Visual Prompting for Efficient Vision-Language Fine-TuningShibo Jie, Yehui Tang, Ning Ding, Zhi-Hong Deng et al.ICML 2024 · 24 citations
- Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsRobert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni et al.ACL Findings 2022 · 133 citations
- PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase GenerationYixin Wan, Kuan-Hao Huang, Kai-Wei ChangACL Findings 2023 · 3 citations
