ACL Findings2021
Gaussian Process based Deep Dyna-Q approach for Dialogue Policy Learning
Guanlin Wu, Wenqi Fang, Ji Wang, Jiang Cao, Weidong Bao, Yang Ping, Xiaomin Zhu, Zheng Wang
Abstract
Applying reinforcement learning to dialogue policy learning requires prohibitively large rounds of human-machine interactions. To improve the learning performance, the Deep Dyna-Q framework with a world model that imitates real users is widely used in recent years. Unfortunately, how to build an effective world model and how to evaluate the experiences generated by the world model efficiently have not been well studied. In order to further improve the effectiveness and efficiency of dialogue policy learning, we present a novel Gaussian Process based Deep Dyna-Q approach in this paper. The Gaussian Process model, which is analytically tractable and fits for small-sample problems, is introduced to build the world model. In addition, we design a highly efficient Kullback-Leibler divergence based discriminator to evaluate the quality of experiences generated by the world model. Extensive experiments validate the effectiveness and robustness of our proposed approach. The task-completion success rate can be improved by about 20% with fewer human-machine interactions.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on4
- Bayesian Meta-Learning for the Few-Shot Setting via Deep KernelsMassimiliano Patacchiola, Jack Turner, Elliot J. Crowley, Michael F. P. O'Boyle et al.NeurIPS 2020 · 167 citations
- Dynamic Reward-Based Dueling Deep Dyna-Q: Robust Policy Learning in Noisy EnvironmentsYangyang Zhao, Zhenyu Wang, Kai Yin, Rui Zhang et al.AAAI 2020 · 32 citations
- Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue SystemsZiming Li, Julia Kiseleva, Maarten de RijkeEMNLP Findings 2020 · 21 citations
- Learning Efficient Dialogue Policy from Demonstrations through ShapingHuimin Wang, Baolin Peng, Kam-Fai WongACL 2020 · 18 citations
Related papers
- Automatic Curriculum Learning With Over-repetition Penalty for Dialogue Policy LearningYangyang Zhao, Zhenyu Wang, Zhenhua HuangAAAI 2021 · 20 citations
- Guided Dialogue Policy Learning without Adversarial Learning in the LoopZiming Li, Sungjin Lee, Baolin Peng, Jinchao Li et al.EMNLP Findings 2020 · 18 citations
- Multi-Agent Task-Oriented Dialog Policy Learning with Role-Aware Reward DecompositionRyuichi Takanobu, Runze Liang, Minlie HuangACL 2020 · 47 citations
- Efficient Dialog Policy Learning by Reasoning with Contextual KnowledgeHaodi Zhang, Zhichao Zeng, Keting Lu, Kaishun Wu et al.AAAI 2022 · 14 citations
- [CASPI] Causal-aware Safe Policy Improvement for Task-oriented DialogueGovardana Sachithanandam Ramachandran, Kazuma Hashimoto, Caiming XiongACL 2022 · 12 citations
