Lune

ACL Findings2024

Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk

Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Lijia Sun, Xibin Gao, Yi Zhang

Abstract

Large language models (LLMs) are powerful dialogue agents, but specializing them towards fulfilling a specific function can be challenging. Instructing tuning, i.e. tuning models on instruction and sample responses generated by humans (Ouyang et al., 2022), has proven as an effective method to do so, yet requires a number of data samples that a) might not be available or b) costly to generate. Furthermore, this cost increases when the goal is to make the LLM follow a specific workflow within a dialogue instead of single instructions. Inspired by the self-play technique in reinforcement learning and the use of LLMs to simulate human agents, we propose a more effective method for data collection through LLMs engaging in a conversation in various roles. This approach generates a training data via "self-talk" of LLMs that can be refined and utilized for supervised fine-tuning. We introduce an automated way to measure the (partial) success of a dialogue. This metric is used to filter the generated conversational data that is fed back in LLM for training. Based on our automated and human evaluations of conversation quality, we demonstrate that such self-talk data improves results. In addition, we examine the various characteristics that showcase the quality of generated dialogues and how they can be connected to their potential utility as training data.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

15 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification cites Training language models to follow instructions with human feedback
  • Generative Agents: Interactive Simulacra of Human Behavior cites Training language models to follow instructions with human feedback
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Systematic Biases in LLM Simulations of Debates cites LoRA: Low-Rank Adaptation of Large Language Models
  • Systematic Biases in LLM Simulations of Debates cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Systematic Biases in LLM Simulations of Debates cites Training language models to follow instructions with human feedback
  • Why Less is More (Sometimes): A Theory of Data Curation cites Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification
  • Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers15

  1. Systematic Biases in LLM Simulations of DebatesAmir Taubenfeld, Yaniv Dover, Roi Reichart, Ariel GoldsteinEMNLP 2024 · 35 citations
  2. EvoChart: A Benchmark and a Self-Training Approach Towards Real-World Chart UnderstandingMuye Huang, Han Lai, Xinyu Zhang, Wenjun Wu et al.AAAI 2025 · 30 citations
  3. Why Less is More (Sometimes): A Theory of Data CurationElvis Dohmatob, Mohammad Pezeshki, Reyhane Askari HemmatICLR 2026 · 11 citations
  4. Watch Every Step! LLM Agent Learning via Iterative Step-level Process RefinementWeimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu et al.EMNLP 2024 · 8 citations
  5. Beyond Model Collapse: Scaling Up with Synthesized Data Requires VerificationYunzhen Feng, Elvis Dohmatob, Pu Yang, François Charton et al.ICLR 2025 · 6 citations

Builds on11

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  5. Generative Agents: Interactive Simulacra of Human BehaviorJoon Sung Park, Joseph C. O'Brien, Carrie Jun Cai, Meredith Ringel Morris et al.UIST 2023 · 1,882 citations

Related papers

  1. Inductive-Deductive Strategy Reuse for Multi-Turn Instructional DialoguesJiao Ou, Jiayu Wu, Che Liu, Fuzheng Zhang et al.EMNLP 2024 · 2 citations
  2. Simulated Students in Tutoring Dialogues: Substance or Illusion?Alexander Scarlatos, Jaewook Lee, Simon Woodhead, Andrew LanACL 2026 · 6 citations
  3. Multiagent Finetuning: Self Improvement with Diverse Reasoning ChainsVighnesh Subramaniam, Yilun Du, Joshua B. Tenenbaum, Antonio Torralba et al.ICLR 2025
  4. Plug-and-Play Policy Planner for Large Language Model Powered Dialogue AgentsYang Deng, Wenxuan Zhang, Wai Lam, See-Kiong Ng et al.ICLR 2024 · 86 citations
  5. Enhancing Chat Language Models by Scaling High-quality Instructional ConversationsNing Ding, Yulin Chen, Bokai Xu, Yujia Qin et al.EMNLP 2023 · 95 citations
Dusk over the sea between two cliffs drawn in fine vertical lines