Lune

ACL Findings2021

Gaussian Process based Deep Dyna-Q approach for Dialogue Policy Learning

Guanlin Wu, Wenqi Fang, Ji Wang, Jiang Cao, Weidong Bao, Yang Ping, Xiaomin Zhu, Zheng Wang

Abstract

Applying reinforcement learning to dialogue policy learning requires prohibitively large rounds of human-machine interactions. To improve the learning performance, the Deep Dyna-Q framework with a world model that imitates real users is widely used in recent years. Unfortunately, how to build an effective world model and how to evaluate the experiences generated by the world model efficiently have not been well studied. In order to further improve the effectiveness and efficiency of dialogue policy learning, we present a novel Gaussian Process based Deep Dyna-Q approach in this paper. The Gaussian Process model, which is analytically tractable and fits for small-sample problems, is introduced to build the world model. In addition, we design a highly efficient Kullback-Leibler divergence based discriminator to evaluate the quality of experiences generated by the world model. Extensive experiments validate the effectiveness and robustness of our proposed approach. The task-completion success rate can be improved by about 20% with fewer human-machine interactions.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on4

  1. Bayesian Meta-Learning for the Few-Shot Setting via Deep KernelsMassimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael F. P. O'Boyle et al.NeurIPS 2020 · 167 citations
  2. Dynamic Reward-Based Dueling Deep Dyna-Q: Robust Policy Learning in Noisy EnvironmentsYangyang Zhao, Zhenyu Wang, Kai Yin, Rui Zhang et al.AAAI 2020 · 32 citations
  3. Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue SystemsZiming Li, Julia Kiseleva, Maarten de RijkeEMNLP Findings 2020 · 21 citations
  4. Learning Efficient Dialogue Policy from Demonstrations through ShapingHuimin Wang, Baolin Peng, Kam-Fai WongACL 2020 · 18 citations

Related papers

  1. Automatic Curriculum Learning With Over-repetition Penalty for Dialogue Policy LearningYangyang Zhao, Zhenyu Wang, Zhenhua HuangAAAI 2021 · 20 citations
  2. Guided Dialogue Policy Learning without Adversarial Learning in the LoopZiming Li, Sungjin Lee, Baolin Peng, Jinchao Li et al.EMNLP Findings 2020 · 18 citations
  3. Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward DecompositionRyuichi Takanobu, Runze Liang, Minlie HuangACL 2020 · 47 citations
  4. Efficient Dialog Policy Learning by Reasoning with Contextual KnowledgeHaodi Zhang, Zhichao Zeng, Keting Lu, Kaishun Wu et al.AAAI 2022 · 14 citations
  5. [CASPI] Causal-aware Safe Policy Improvement for Task-oriented DialogueGovardana Sachithanandam Ramachandran, Kazuma Hashimoto, Caiming XiongACL 2022 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines