Lune

EMNLP Findings2022

Efficient (Soft) Q-Learning for Text Generation with Limited Good Data

Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, Zhiting Hu

Abstract

Maximum likelihood estimation (MLE) is the predominant algorithm for training text generation models. This paradigm relies on direct supervision examples, which is not applicable to many emerging applications, such as generating adversarial attacks or generating prompts to control language models. Reinforcement learning (RL) on the other hand offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. Yet previous RL algorithms for text generation, such as policy gradient (on-policy RL) and Q-learning (off-policy RL), are often notoriously inefficient or unstable to train due to the large sequence space and the sparse reward received only at the end of sequences. In this paper, we introduce a new RL formulation for text generation from the soft Q-learning (SQL) perspective. It enables us to draw from the latest RL advances, such as path consistency learning, to combine the best of on-/off-policy updates, and learn effectively from sparse reward. We apply the approach to a wide range of novel text generation tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation. Experiments show our approach consistently outperforms both task-specialized algorithms and the previous RL methods. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Plug and Play Language Models: A Simple Approach to Controlled Text Generation cites The Curious Case of Neural Text Degeneration
  • Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo cites Plug and Play Language Models: A Simple Approach to Controlled Text Generation
  • Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo cites Amortizing intractable inference in large language models
  • Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo cites The Curious Case of Neural Text Degeneration
  • Amortizing intractable inference in large language models cites The Curious Case of Neural Text Degeneration
  • Preference-grounded Token-level Guidance for Language Model Fine-tuning cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • Preference-grounded Token-level Guidance for Language Model Fine-tuning cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language ModelsHaonan Duan, Adam Dziedzic, Nicolas Papernot, Franziska BoenischNeurIPS 2023 · 116 citations
  2. Amortizing intractable inference in large language modelsEdward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar et al.ICLR 2024 · 91 citations
  3. When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided SearchXuan Chen, Yuzhou Nie, Wenbo Guo, Xiangyu ZhangNeurIPS 2024 · 68 citations
  4. Probabilistic Inference in Language Models via Twisted Sequential Monte CarloStephen Zhao, Rob Brekelmans, Alireza Makhzani, Roger Baker GrosseICML 2024 · 61 citations
  5. Preference-grounded Token-level Guidance for Language Model Fine-tuningShentao Yang, Shujian Zhang, Congying Xia, Yihao Feng et al.NeurIPS 2023 · 39 citations

Builds on18

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  2. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
  3. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
  4. Plug and Play Language Models: A Simple Approach to Controlled Text GenerationSumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung et al.ICLR 2020 · 1,166 citations
  5. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations

Related papers

  1. Imitating Language via Scalable Inverse Reinforcement LearningMarkus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja et al.NeurIPS 2024 · 26 citations
  2. Teacher Forcing Recovers Reward Functions for Text GenerationYongchang Hao, Yuxin Liu, Lili MouNeurIPS 2022 · 24 citations
  3. Offline RL for Natural Language Generation with Implicit Language Q LearningCharlie Snell, Ilya Kostrikov, Yi Su, Sherry Yang et al.ICLR 2023 · 9 citations
  4. Text Generation by Learning from DemonstrationsRichard Yuanzhe Pang, He HeICLR 2021 · 88 citations
  5. Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashionYannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Eugene Choi et al.EMNLP 2024
Dusk over the sea between two cliffs drawn in fine vertical lines