ACL Findings2021
Addressing Inquiries about History: An Efficient and Practical Framework for Evaluating Open-domain Chatbot Consistency
Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, Jie Zhou
Abstract
A good open-domain chatbot should avoid presenting contradictory responses about facts or opinions in a conversational session, known as its consistency capacity. However, evaluating the consistency capacity of a chatbot is still challenging. Employing human judges to interact with chatbots on purpose to check their capacities is costly and low-efficient, and difficult to get rid of subjective bias. In this paper, we propose the Addressing Inquiries about History (AIH), an efficient and practical framework for the consistency evaluation. At the conversation stage, AIH attempts to address appropriate inquiries about the dialogue history to induce the chatbot to redeclare the historical facts or opinions. We carry out the conversation between chatbots, which is more efficient than the human-bot interaction and can also alleviate the subjective bias. In this way, we manage to rapidly obtain a dialog session that contains responses with high contradiction possibilities. At the contradiction recognition stage, we can either employ human judges or a natural language inference (NLI) model to recognize whether the answers to the inquiries are contradictory with history. Finally, we are able to rank chatbots according to the contradiction statistics. Experiments on open-domain chatbots show that our approach can efficiently and reliably assess the consistency capacity of chatbots and achieve a high ranking correlation with the human evaluation. We release the framework and hope to help improve the consistency capacity of chatbots. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems cites Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills
- PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning cites Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skills
- PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning cites Language Models are Few-Shot Learners
- CDConv: A Benchmark for Contradiction Detection in Chinese Conversations cites Profile Consistency Identification for Open-domain Dialogue Agents
- Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
Cited by top-tier papers2
- Don't Forget Your ABC's: Evaluating the State-of-the-Art in Chat-Oriented Dialogue SystemsSarah E. Finch, James D. Finch, Jinho D. ChoiACL 2023 · 10 citations
- CDConv: A Benchmark for Contradiction Detection in Chinese ConversationsChujie Zheng, Jinfeng Zhou, Yinhe Zheng, Libiao Peng et al.EMNLP 2022 · 6 citations
Builds on6
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum LearningSiqi Bao, Huang He, Fan Wang, Hua Wu et al.ACL Findings 2021 · 94 citations
- Profile Consistency Identification for Open-domain Dialogue AgentsHaoyu Song, Yan Wang, Wei-Nan Zhang, Zhengyu Zhao et al.EMNLP 2020 · 19 citations
- Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend SkillsEric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston et al.ACL 2020 · 18 citations
Related papers
- Spot The Bot: A Robust and Efficient Framework for the Evaluation of Conversational Dialogue SystemsJan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos et al.EMNLP 2020 · 12 citations
- I like fish, especially dolphins: Addressing Contradictions in Dialogue ModelingYixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela et al.ACL 2021
- Towards Holistic and Automatic Evaluation of Open-Domain Dialogue GenerationBo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou et al.ACL 2020 · 69 citations
- Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue SystemLibo Qin, Tianbao Xie, Shijue Huang, Qiguang Chen et al.EMNLP 2021 · 9 citations
- Interview Evaluation: A Novel Approach for Automatic Evaluation of Conversational Question Answering ModelsXibo Li, Bowei Zou, Yifan Fan, Yanling Li et al.EMNLP 2023
