EMNLP Findings2020
What do we expect from Multiple-choice QA Systems?
Abstract
The recent success of machine learning systems on various QA datasets could be interpreted as a significant improvement in models' language understanding abilities. However, using various perturbations, multiple recent works have shown that good performance on a dataset might not indicate performance that correlates well with human's expectations from models that "understand" language. In this work we consider a top performing model on several Multiple Choice Question Answering (MCQA) datasets, and evaluate it against a set of expectations one might have from such a model, using a series of zero-information perturbations of the model's inputs. Our results show that the model clearly falls short of our expectations, and motivates a modified training approach that forces the model to better attend to the inputs. We show that the new training paradigm leads to a model that performs on par with the original model while better satisfying our expectations. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
- Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?Nishant Balepur, Abhilasha Ravichander, Rachel RudingerACL 2024 · 5 citations
- DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence UnderstandingXiaoyu Guo, Yuan-Fang Li, Gholamreza HaffariEMNLP 2023 · 1 citation
Builds on4
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
- Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
- QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations
- Good-Enough Compositional Data AugmentationJacob AndreasACL 2020 · 15 citations
Related papers
- Enhancing Multiple-choice Machine Reading Comprehension by Punishing Illogical InterpretationsYiming Ju, Yuanzhe Zhang, Zhixing Tian, Kang Liu et al.EMNLP 2021 · 8 citations
- Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice QuestionsSarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi et al.ICLR 2025
- LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum LearningSayan Ghosh, Rakesh R. Menon, Shashank SrivastavaACL Findings 2023 · 2 citations
- Assessing the Benchmarking Capacity of Machine Reading Comprehension DatasetsSaku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko AizawaAAAI 2020 · 92 citations
- Leveraging Large Language Models for Multiple Choice Question AnsweringJoshua Robinson, David WingateICLR 2023 · 40 citations
