Lune

ACL Findings2023

Attribute Controlled Dialogue Prompting

Runcheng Liu, Ahmad Rashid, Ivan Kobyzev, Mehdi Rezagholizadeh, Pascal Poupart

Abstract

Prompt-tuning has become an increasingly popular parameter-efficient method for adapting large pretrained language models to downstream tasks.However, both discrete prompting and continuous prompting assume fixed prompts for all data samples within a task, neglecting the fact that inputs vary greatly in some tasks such as open-domain dialogue generation.In this paper, we present a novel, instancespecific prompt-tuning algorithm for dialogue generation.Specifically, we generate prompts based on instance-level control code, rather than the conversation history, to explore their impact on controlled dialogue generation.Experiments on popular open-domain dialogue datasets, evaluated on both automated metrics and human evaluation, demonstrate that our method is superior to prompting baselines and comparable to fine-tuning with only 5%-6% of total parameters. overhead.We present results on both intent and persona controlled dialogue.2 Related Work GPT-3 (Brown et al., 2020) introduces prompting, a method to steer a frozen PLM by transforming inputs into cloze-style phrases with task description and some task examples.Though it is memoryefficient since one single copy of the PLM can be shared across different tasks, the model's performance is largely restricted by the maximum conditional input length, the model size and manual guesswork for prompts (Zhao et al., 2021; Schick and Schütze, 2021a,b;Jiang et al., 2020).Other works focus on automatically searching for better discrete prompts (Jiang et al., 2020;Shin et al., 2020;Gao et al., 2021;Ben-David et al., 2021).Recently, there has been an increased interest in continuous prompts / prompt-tuning, which bridges the gap between prompting and fine-tuning, while remaining efficient during training (Lester et al., 2021;Li and Liang, 2021;Liu et al., 2021Liu et al., , 2022)).Continuous prompts extend prompt selection to the entire space of embeddings, including vector embeddings that do not correspond to any humaninterpretable natural language tokens.Hence, soft prompts are more expressive than discrete prompts.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • HyperPrompt: Prompt-based Task-Conditioning of Transformers cites The Power of Scale for Parameter-Efficient Prompt Tuning
  • HyperPrompt: Prompt-based Task-Conditioning of Transformers cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • HyperPrompt: Prompt-based Task-Conditioning of Transformers cites Language Models are Few-Shot Learners
  • The Power of Scale for Parameter-Efficient Prompt Tuning cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Towards Cost-Effective Reward Guided Text GenerationAhmad Rashid, Ruotian Wu, Rongqi Fan, Hongliang Li et al.ICML 2025

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  3. HyperPrompt: Prompt-based Task-Conditioning of TransformersYun He, Huaixiu Steven Zheng, Yi Tay, Jai Prakash Gupta et al.ICML 2022 · 110 citations
  4. The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
  5. Controllable Natural Language Generation with Contrastive PrefixesJing Qian, Li Dong, Yelong Shen, Furu Wei et al.ACL Findings 2022 · 74 citations

Related papers

  1. Late Prompt Tuning: A Late Prompt Could Be Better Than Many PromptsXiangyang Liu, Tianxiang Sun, Xuanjing Huang, Xipeng QiuEMNLP Findings 2022 · 15 citations
  2. Selective Prompting Tuning for Personalized Conversations with LLMsQiushi Huang, Xubo Liu, Tom Ko, Bo Wu et al.ACL Findings 2024 · 12 citations
  3. ADPL: Adversarial Prompt-based Domain Adaptation for Dialogue Summarization with Knowledge DisentanglementLulu Zhao, Fujia Zheng, Weihao Zeng, Keqing He et al.SIGIR 2022 · 6 citations
  4. MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading ComprehensionGuoxin Chen, Yiming Qian, Bowen Wang, Liangzhi LiEMNLP Findings 2023 · 6 citations
  5. APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language ModelsQifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu et al.EMNLP 2023 · 25 citations
Dusk over the sea between two cliffs drawn in fine vertical lines