EMNLP Findings2023
Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, Luke Zettlemoyer
Abstract
Large language models can perform downstream tasks in a zero-shot fashion, given natural language prompts that specify the desired behavior.Such prompts are typically hand engineered, but can also be learned with gradientbased methods from labeled data.However, it is underexplored what factors make the prompts effective, especially when the prompts are in natural language.In this paper, we investigate common attributes shared by effective prompts in classification problems.We first propose a human readable prompt tuning method (FLUENTPROMPT) based on Langevin dynamics that incorporates a fluency constraint to find a distribution of effective and fluent prompts.Our analysis reveals that effective prompts are topically related to the task domain and calibrate the prior probability of output labels.Based on these findings, we also propose a method for generating prompts using only unlabeled data, outperforming strong baselines by an average of 7.0% accuracy across three tasks.We release our code and data in github.com/swj0419/FluentPrompt. Order randomly determined.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers cites Large Language Models are Zero-Shot Reasoners
- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers cites Language Models are Few-Shot Learners
- Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Language Models are Few-Shot Learners
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Calibrate Before Use: Improving Few-shot Performance of Language Models
- Large Language Models are Zero-Shot Reasoners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
- Large Language Models are Zero-Shot Reasoners cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts cites Language Models are Few-Shot Learners
- Putting People in LLMs' Shoes: Generating Better Answers via Question Rewriter cites Large Language Models are Zero-Shot Reasoners
- SPRIG: Improving Large Language Model Performance by System Prompt Optimization cites Language Models are Few-Shot Learners
- Calibrate Before Use: Improving Few-shot Performance of Language Models cites Language Models are Few-Shot Learners
Cited by top-tier papers11
- Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic PromptsZhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen et al.ICML 2024 · 155 citations
- SPRIG: Improving Large Language Model Performance by System Prompt OptimizationLechen Zhang, Tolga Ergen, Lajanugen Logeswaran, Moontae Lee et al.ICLR 2026 · 35 citations
- Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with TransformersXiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu et al.ICML 2024 · 26 citations
- PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel OptimizationYang Jiao, Xiaodong Wang, Kai YangSIGIR 2025 · 6 citations
- Putting People in LLMs' Shoes: Generating Better Answers via Question RewriterJunhao Chen, Bowen Wang, Zhouqiang Jiang, Yuta NakashimaAAAI 2025 · 5 citations
Builds on16
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
- Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
Related papers
- Large Language Models as OptimizersChengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu et al.ICLR 2024 · 817 citations
- Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language ModelsZijun Wu, Yongkang Wu, Lili MouICLR 2024 · 9 citations
- PPT: Pre-trained Prompt Tuning for Few-shot LearningYuxian Gu, Xu Han, Zhiyuan Liu, Minlie HuangACL 2022
- Two Examples are Better than One: Context Regularization for Gradient-based Prompt TuningHyeonmin Ha, Soyoung Jung, Jinsol Park, Minjoon Seo et al.ACL Findings 2023 · 1 citation
- Large Language Models are Human-Level Prompt EngineersYongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster et al.ICLR 2023 · 297 citations
