Lune

ACL Findings2023

Fixed Input Parameterization for Efficient Prompting

Eunbi Choi, Yongrae Jo, Joel Jang, Joonwon Jang, Minjoon Seo

摘要

Recent works have shown that attaching prompts to the input is effective at conditioning Language Models (LM) to perform specific tasks. However, prompts are always included in the input text during inference, even when they are fixed, thus incurring substantial computational and memory overhead. Also, there is currently no straightforward method of utilizing prompts that are longer than the maximum input length of the LMs without incurring additional costs during inference. We formally define Fixed Input Parameterization (FIP) problem that focuses on injecting the fixed prompt into the parameters of an LM to be an efficient alternative to attaching fixed prompts to the input. We show that in scenarios with long fixed prompts, FIP can be up to 280 times more efficient in terms of total FLOPs than previous approaches. We further explore methodologies for FIP and show promising results in persona-dependent conversation, semantic parsing, and zero-shot learning with task instructions. Through these explorations, we show that FIP can be a promising direction for conditioning language models, in scenarios with long and fixed prompts 1 .

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • LoRA: Low-Rank Adaptation of Large Language Models 引用了 Language Models are Few-Shot Learners
  • Doc-to-LoRA: Learning to Instantly Internalize Contexts 引用了 Language Models are Few-Shot Learners
  • Multitask Prompted Training Enables Zero-Shot Task Generalization 引用了 Language Models are Few-Shot Learners
  • Multitask Prompted Training Enables Zero-Shot Task Generalization 引用了 Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners 引用了 Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Finetuned Language Models are Zero-Shot Learners 引用了 Language Models are Few-Shot Learners
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper1

  1. Doc-to-LoRA: Learning to Instantly Internalize ContextsRujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert LangeICML 2026 · 27 次引用

它引用的 Paper12

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
  3. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu 等ICLR 2022 · 4,966 次引用
  4. Transformers are RNNs: Fast Autoregressive Transformers with Linear AttentionAngelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François FleuretICML 2020 · 2,665 次引用
  5. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach 等ICLR 2022 · 1,976 次引用

相关 Paper

  1. PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningJiaru Zou, Mengyu Zhou, Tao Li, Shi Han 等EMNLP Findings 2024 · 6 次引用
  2. Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction FollowingSeonghyeon Ye, Hyeonbin Hwang, Sohee Yang, Hyeongu Yun 等AAAI 2024 · 50 次引用
  3. Memory-Space Visual Prompting for Efficient Vision-Language Fine-TuningShibo Jie, Yehui Tang, Ning Ding, Zhi-Hong Deng 等ICML 2024 · 24 次引用
  4. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language ModelsRobert L. Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni 等ACL Findings 2022 · 133 次引用
  5. PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase GenerationYixin Wan, Kuan-Hao Huang, Kai-Wei ChangACL Findings 2023 · 3 次引用
黄昏的海面,两侧是细线勾勒的悬崖