ACL Findings2021
GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation
Hongye Tan, Xiaoyue Wang, Yu Ji, Ru Li, Xiaoli Li, Zhiwei Hu, Yunxiao Zhao, Xiaoqi Han
Abstract
Recently, driven by numerous publicly available machine reading comprehension (MRC) datasets, MRC systems have made some progress. These datasets, however, have two major limitations: 1) the defined tasks are relatively simple, and 2) they do not provide explainable evaluation which is critical to objectively and comprehensively review the reasoning capabilities of current MRC systems. In this paper, we propose GCRC, a new dataset with challenging and high-quality multi-choice questions, collected from Gaokao Chinese (Chinese subject from the National College Entrance Examination of China). We have manually labelled three types of evidence to evaluate MRC systems' reasoning process: 1) sentence-level relevant supporting facts in an article required for answering a given question, 2) error reason of a distractor (i.e., an incorrect option) for explaining why a distractor should be eliminated, which is an important reasoning step for multi-choice questions, and 3) types of reasoning skills required for answering questions. Extensive experiments show that our proposed dataset is more challenging and very useful for identifying the limitations of existing MRC systems in an explainable way, facilitating researchers to develop novel machine learning and reasoning approaches to tackle this challenging research problem. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on1
Related papers
- ReCO: A Large Scale Chinese Reading Comprehension Dataset on OpinionBingning Wang, Ting Yao, Qi Zhang, Jingfang Xu et al.AAAI 2020 · 26 citations
- Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering DataDian Yu, Kai Sun, Dong Yu, Claire CardieEMNLP Findings 2021 · 2 citations
- Extract, Integrate, Compete: Towards Verification Style Reading ComprehensionChen Zhang, Yuxuan Lai, Yansong Feng, Dongyan ZhaoEMNLP Findings 2021 · 7 citations
- Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading ComprehensionAkira Kawabata, Saku SugawaraEMNLP 2023
- MMM: Multi-Stage Multi-Task Learning for Multi-Choice Reading ComprehensionDi Jin, Shuyang Gao, Jiun-Yu Kao, Tagyoung Chung et al.AAAI 2020 · 72 citations
