EMNLP Findings2023
Machine Reading Comprehension using Case-based Reasoning
Dung Thai, Dhruv Agarwal, Mudit Chaudhary, Wenlong Zhao, Rajarshi Das, Jay-Yoon Lee, Hannaneh Hajishirzi, Manzil Zaheer, Andrew McCallum
Abstract
We present an accurate and interpretable method for answer extraction in machine reading comprehension that is reminiscent of casebased reasoning (CBR) from classical AI. Our method (CBR-MRC) builds upon the hypothesis that contextualized answers to similar questions share semantic similarities with each other. Given a test question, CBR-MRC first retrieves a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers in the retrieved cases. The semi-parametric nature of our approach allows it to attribute a prediction to the specific set of evidence cases, making it a desirable choice for building reliable and debuggable QA systems. We show that CBR-MRC provides high accuracy comparable with large reader models and outperforms baselines by 11.5 and 8.4 EM on NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability of CBR-MRC in identifying not just the correct answer tokens but also the span with the most relevant supporting evidence. Lastly, we observe that contexts for certain question types show higher lexical diversity than others and find that CBR-MRC is robust to these variations while performance using fully-parametric methods drops. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- A Statistical Framework for Data-dependent Retrieval-Augmented Models cites Language Models are Few-Shot Learners
- A Statistical Framework for Data-dependent Retrieval-Augmented Models cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Language Models are Few-Shot Learners cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Cited by top-tier papers1
Builds on16
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
Related papers
- Case-based Reasoning for Natural Language Queries over Knowledge BasesRajarshi Das, Manzil Zaheer, Dung Thai, Ameya Godbole et al.EMNLP 2021 · 111 citations
- Generating Extractive Answers: Gated Recurrent Memory Reader for Conversational Question AnsweringXuanyu Zhang, Qing YangEMNLP Findings 2023
- A Self-Training Method for Machine Reading Comprehension with Soft Evidence ExtractionYilin Niu, Fangkai Jiao, Mantong Zhou, Ting Yao et al.ACL 2020 · 33 citations
- REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-trainingFangkai Jiao, Yangyang Guo, Yilin Niu, Feng Ji et al.ACL Findings 2021 · 8 citations
- Recurrent Chunking Mechanisms for Long-Text Machine Reading ComprehensionHongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen et al.ACL 2020 · 39 citations
