Lune

EMNLP Findings2020

Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

Carel van Niekerk, Michael Heck, Christian Geishauser, Hsien-Chin Lin, Nurul Lubis, Marco Moresi, Milica Gasic

Abstract

The ability to accurately track what happens during a conversation is essential for the performance of a dialogue system. Current stateof-the-art multi-domain dialogue state trackers achieve just over 55% accuracy on the current go-to benchmark, which means that in almost every second dialogue turn they place full confidence in an incorrect dialogue state. Belief trackers, on the other hand, maintain a distribution over possible dialogue states. However, they lack in performance compared to dialogue state trackers, and do not produce well calibrated distributions. In this work we present state-of-the-art performance in calibration for multi-domain dialogue belief trackers using a calibrated ensemble of models. Our resulting dialogue belief tracker also outperforms previous dialogue belief tracking models in terms of accuracy.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance cites Ensemble Distribution Distillation
  • A Simple Language Model for Task-Oriented Dialogue cites Schema-Guided Multi-Domain Dialogue State Tracking with Graph Attention Neural Networks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy PerformanceCarel van Niekerk, Andrey Malinin, Christian Geishauser, Michael Heck et al.EMNLP 2021 · 8 citations
  2. On the Calibration and Uncertainty with Pólya-Gamma Augmentation for Dialog Retrieval ModelsTong Ye, Shijing Si, Jianzong Wang, Ning Cheng et al.AAAI 2023 · 3 citations
  3. Belief Revision: The Adaptability of Large Language Models ReasoningBryan Wilie, Samuel Cahyawijaya, Etsuko Ishii, Junxian He et al.EMNLP 2024

Builds on5

  1. A Simple Language Model for Task-Oriented DialogueEhsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz et al.NeurIPS 2020 · 590 citations
  2. Ensemble Distribution DistillationAndrey Malinin, Bruno Mlodozeniec, Mark J. F. GalesICLR 2020 · 273 citations
  3. Efficient Dialogue State Tracking by Selectively Overwriting MemorySungdong Kim, Sohee Yang, Gyuwan Kim, Sang-Woo LeeACL 2020 · 189 citations
  4. Schema-Guided Multi-Domain Dialogue State Tracking with Graph Attention Neural NetworksLu Chen, Boer Lv, Chi Wang, Su Zhu et al.AAAI 2020 · 143 citations
  5. Being Bayesian about Categorical ProbabilityTaejong Joo, Uijung Chung, Min-Gwan SeoICML 2020 · 69 citations

Related papers

  1. GCDST: A Graph-based and Copy-augmented Multi-domain Dialogue State TrackingPeng Wu, Bowei Zou, Ridong Jiang, AiTi AwEMNLP Findings 2020 · 13 citations
  2. Multi-Domain Dialogue State Tracking with Disentangled Domain-Slot AttentionLongfei Yang, Jiyi Li, Sheng Li, Takahiro ShinozakiACL Findings 2023 · 5 citations
  3. Similarity-based Multi-Domain Dialogue State Tracking with Copy Mechanisms for Task-based Virtual Personal AssistantsJarana Manotumruksa, Jeffrey Dalton, Edgar Meij, Emine YilmazWWW 2022 · 5 citations
  4. Correctable-DST: Mitigating Historical Context Mismatch between Training and Inference for Improved Dialogue State TrackingHongyan Xie, Haoxiang Su, Shuangyong Song, Hao Huang et al.EMNLP 2022 · 10 citations
  5. Multi-domain Dialogue State Tracking with Recursive InferenceLizi Liao, Tongyao Zhu, Le Hong Long, Tat-Seng ChuaWWW 2021 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines