Lune

EMNLP Findings2020

Joint Turn and Dialogue level User Satisfaction Estimation on Mulit-Domain Conversations

Praveen Kumar Bodigutla, Aditya Tiwari, Spyros Matsoukas, Josep Valls-Vargas, Lazaros Polymenakos

Abstract

Dialogue level quality estimation is vital for optimizing data driven dialogue management. Current automated methods to estimate turn and dialogue level user satisfaction employ hand-crafted features and rely on complex annotation schemes, which reduce the generalizability of the trained models. We propose a novel user satisfaction estimation approach which minimizes an adaptive multi-task loss function in order to jointly predict turn-level Response Quality labels provided by experts and explicit dialogue-level ratings provided by end users. The proposed BiLSTM based deep neural net model automatically weighs each turn's contribution towards the estimated dialogue-level rating, implicitly encodes temporal dependencies, and removes the need to hand-craft features.

On dialogues sampled from 28 Alexa domains, two dialogue systems and three user groups, the joint dialogue-level satisfaction estimation model achieved up to an absolute 27% (0.43 → 0.70) and 7% (0.63 → 0.70) improvement in linear correlation performance over baseline deep neural net and benchmark Gradient boosting regression models, respectively.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • A Speaker Turn-Aware Multi-Task Adversarial Network for Joint User Satisfaction Estimation and Sentiment Analysis cites User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems
  • Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process cites User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational SystemsYang Deng, Wenxuan Zhang, Wai Lam, Hong Cheng et al.WWW 2022 · 34 citations
  2. A Role-Selected Sharing Network for Joint Machine-Human Chatting Handoff and Service Satisfaction AnalysisJiawei Liu, Kaisong Song, Yangyang Kang, Guoxiu He et al.EMNLP 2021 · 7 citations
  3. Modeling User Satisfaction Dynamics in Dialogue via Hawkes ProcessFanghua Ye, Zhiyuan Hu, Emine YilmazACL 2023 · 6 citations
  4. A Speaker Turn-Aware Multi-Task Adversarial Network for Joint User Satisfaction Estimation and Sentiment AnalysisKaisong Song, Yangyang Kang, Jiawei Liu, Xurui Li et al.AAAI 2023 · 5 citations
  5. Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMsClemencia Siro, Mohammad Aliannejadi, Maarten de RijkeSIGIR 2024 · 3 citations

Related papers

  1. What is wrong with you?: Leveraging User Sentiment for Automatic Dialog EvaluationSarik Ghazarian, Behnam Hedayatnia, Alexandros Papangelis, Yang Liu et al.ACL Findings 2022 · 18 citations
  2. Predictive Engagement: An Efficient Metric for Automatic Evaluation of Open-Domain Dialogue SystemsSarik Ghazarian, Ralph M. Weischedel, Aram Galstyan, Nanyun PengAAAI 2020 · 62 citations
  3. Beyond User Self-Reported Likert Scale Ratings: A Comparison Model for Automatic Dialog EvaluationWeixin Liang, James Zou, Zhou YuACL 2020 · 25 citations
  4. UEQManager: A Non-intrusive and Real-time System for Recognizing and Managing UEQ in Multilingual Voice AssistantsEntong Gao, Haisu Yuan, Hanyu Zhong, Ruiqing Yuan et al.CHI 2026 · 2 citations
  5. FineD-Eval: Fine-grained Automatic Dialogue-Level EvaluationChen Zhang, Luis Fernando D'Haro, Qiquan Zhang, Thomas Friedrichs et al.EMNLP 2022 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines