MuTual: A Dataset for Multi-Turn Dialogue Reasoning

Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce Mu-Tual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems. Empirical results show that state-of-the-art methods only reach 71%, which is far behind the human performance of 94%, indicating that there is ample room for improving reasoning ability. MuTual is available at https://github. com/Nealcly/MuTual . * Contribution during internship at MSRA. M: Ma'am, you forgot your phone. F: Oh, thanks, I couldn't live without this little thing. M: I know what you mean. It is of great significance to you. So did you enjoy your dinner? F: Oh yes, everything was just perfect. It's so hard to take the whole family out to eat, but your restaurant was perfect. Johnny had his own place to play in and I had time to talk with my sisters and their husbands. ✓ (A) M: Thanks for your compliment for the restaurant. ✘ (B) M: I'm sorry that you don't have a good time. ✘ (C) M: Goodbye brother! Love you. ✘ (D) M: Hurry up honey, or we will be late for the dinner.