Lune

ACL Findings2021

GCRC: A New Challenging MRC Dataset from Gaokao Chinese for Explainable Evaluation

Hongye Tan, Xiaoyue Wang, Yu Ji, Ru Li, Xiaoli Li, Zhiwei Hu, Yunxiao Zhao, Xiaoqi Han

Abstract

Recently, driven by numerous publicly available machine reading comprehension (MRC) datasets, MRC systems have made some progress. These datasets, however, have two major limitations: 1) the defined tasks are relatively simple, and 2) they do not provide explainable evaluation which is critical to objectively and comprehensively review the reasoning capabilities of current MRC systems. In this paper, we propose GCRC, a new dataset with challenging and high-quality multi-choice questions, collected from Gaokao Chinese (Chinese subject from the National College Entrance Examination of China). We have manually labelled three types of evidence to evaluate MRC systems' reasoning process: 1) sentence-level relevant supporting facts in an article required for answering a given question, 2) error reason of a distractor (i.e., an incorrect option) for explaining why a distractor should be eliminated, which is an important reasoning step for multi-choice questions, and 3) types of reasoning skills required for answering questions. Extensive experiments show that our proposed dataset is more challenging and very useful for identifying the limitations of existing MRC systems in an explainable way, facilitating researchers to develop novel machine learning and reasoning approaches to tackle this challenging research problem. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. CARE: A Clue-guided Assistant for CSRs to Read User ManualsWeihong Du, Jia Liu, Zujie Wen, Dingnan Jin et al.ACL 2024

Builds on1

  1. ReClor: A Reading Comprehension Dataset Requiring Logical ReasoningWeihao Yu, Zihang Jiang, Yanfei Dong, Jiashi FengICLR 2020 · 325 citations

Related papers

  1. ReCO: A Large Scale Chinese Reading Comprehension Dataset on OpinionBingning Wang, Ting Yao, Qi Zhang, Jingfang Xu et al.AAAI 2020 · 26 citations
  2. Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering DataDian Yu, Kai Sun, Dong Yu, Claire CardieEMNLP Findings 2021 · 2 citations
  3. Extract, Integrate, Compete: Towards Verification Style Reading ComprehensionChen Zhang, Yuxuan Lai, Yansong Feng, Dongyan ZhaoEMNLP Findings 2021 · 7 citations
  4. Evaluating the Rationale Understanding of Critical Reasoning in Logical Reading ComprehensionAkira Kawabata, Saku SugawaraEMNLP 2023
  5. MMM: Multi-Stage Multi-Task Learning for Multi-Choice Reading ComprehensionDi Jin, Shuyang Gao, Jiun-Yu Kao, Tagyoung Chung et al.AAAI 2020 · 72 citations
Dusk over the sea between two cliffs drawn in fine vertical lines