Lune

EMNLP Findings2023

Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service

Hyeonmin Ha, Jihye Lee, Wookje Han, Byung-Gon Chun

Abstract

Recently, many companies have been providing the capabilities of large language models as services. These Language-Model-as-a-Service (LMaaS) offerings support a variety of user tasks through in-context learning from prompts, which include instructions and demonstrations of the task. However, for users, manually crafting prompts or running automatic prompt tuning methods themselves can be demanding. Despite these challenges, LMaaS providers do not offer automatic prompt engineering methods as part of their services. One of the major obstacles to deploying them on an LMaaS is the heavy computational costs associated with automatic prompt engineering methods. These methods are typically designed to iterate through tens of thousands of examples, which impose unaffordable overheads for LMaaS providers. In this paper, we introduce MetaL-Prompt, a novel lightweight automatic prompt generation method for LMaaS. MetaL-Prompt meta-trains a prompt generation model (PGM) to enable robust learning by the language model from the contexts created by the generated prompts (i.e., in-context learning). Thanks to our meta-learning approach, a PGM can generate prompts for unseen tasks without requiring additional training for those specific tasks. Furthermore, the PGM can generate prompts with a single forward pass, significantly reducing computational costs compared to previous methods. We evaluate MetaL-Prompt on a range of unseen tasks and find that it improves performance by up to 19.4% in terms of mean F1 score on QA datasets compared to the state-of-the-art baseline P-tuning, with limited computational cost.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Calibrate Before Use: Improving Few-shot Performance of Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. CAP: Controllable Alignment Prompting for Unlearning in LLMsZhaokun Wang, Jinyu Guo, Jingwen Pu, Hongli Pu et al.ACL 2026

Builds on16

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
  3. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
  4. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  5. Orca: A Distributed Serving System for Transformer-Based Generative ModelsGyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojeong Kim et al.OSDI 2022 · 690 citations

Related papers

  1. Black-Box Tuning for Language-Model-as-a-ServiceTianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang et al.ICML 2022 · 343 citations
  2. SEE: Strategic Exploration and Exploitation for Cohesive In-Context Prompt OptimizationWendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun et al.ACL 2025
  3. λ-Tune: Harnessing Large Language Models for Automated Database System TuningVictor Giannakouris, Immanuel TrummerSIGMOD 2025 · 20 citations
  4. Prompt-MII: Meta-Learning Instruction Induction for LLMsEmily Xiao, Yixiao Zeng, Ada Chen, Chin-Jou Li et al.ICLR 2026 · 9 citations
  5. Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of ExemplarsZhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu et al.NeurIPS 2024 · 44 citations
Dusk over the sea between two cliffs drawn in fine vertical lines