EMNLP Findings2024
Active Listening: Personalized Question Generation in Open-Domain Social Conversation with User Model Based Prompting
Kevin Bowden, Yue Fan, Winson Chen, Wen Cui, Davan Harrison, Xin Wang, Marilyn A. Walker
Abstract
Large language models (LLMs) capable of casual conversation have recently become widely available. We hypothesize that users of conversational systems want a more personalized experience, and existing work shows that users are highly receptive to personalized questions (PQs). Question Generation tasks, however, focus on factual questions from textual excerpts. To create a PQ generator, we first identify over 400 real user interests by anonymously aggregating ∼39K user models. We then populate prompt templates with these 400 interests and use an LLM to generate PQs customized to user interests. The result is PerQs, a novel corpus of ∼19K question/answer pairs. We evaluate PerQs at scale in the unique context of the Alexa Prize. Our results show significant positive effects on perceived conversation quality. We then fine-tune, deploy, and evaluate PerQy, a neural model that generates PQs in real-time. When evaluated against several competitive LLM baselines, PerQy produced the most natural and engaging responses.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling cites Language Models are Few-Shot Learners
- Designing a Chatbot as a Mediator for Promoting Deep Self-Disclosure to a Real Mental Health Professional cites "I Hear You, I Feel You": Encouraging Deep Self-disclosure through a Chatbot
Builds on10
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
- "I Hear You, I Feel You": Encouraging Deep Self-disclosure through a ChatbotYi-Chieh Lee, Naomi Yamashita, Yun Huang, Wai FuCHI 2020 · 333 citations
- Designing a Chatbot as a Mediator for Promoting Deep Self-Disclosure to a Real Mental Health ProfessionalYi-Chieh Lee, Naomi Yamashita, Yun HuangCSCW 2020 · 190 citations
- Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation ExtractorsKai Zhang, Bernal Jimenez Gutierrez, Yu SuACL Findings 2023 · 69 citations
Related papers
- FreshLLMs: Refreshing Large Language Models with Search Engine AugmentationTu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant et al.ACL Findings 2024 · 73 citations
- Personalized Question Answering with User Profile Generation and CompressionHang Su, Yun Yang, Tianyang Liu, Xin Liu et al.EMNLP Findings 2025
- PersonalLLM: Tailoring LLMs to Individual PreferencesThomas P. Zollo, Andrew Wei Tung Siah, Naimeng Ye, Ang Li et al.ICLR 2025
- Understanding and Supporting Formal Email Exchange by Answering AI-Generated QuestionsYusuke Miura, Chi-Lan Yang, Masaki Kuribayashi, Keigo Matsumoto et al.CHI 2025 · 4 citations
- Are LLMs Aware that Some Questions are not Open-ended?Dongjie Yang, Hai ZhaoEMNLP Findings 2024 · 1 citation
