Lune

EMNLP Findings2020

What do we expect from Multiple-choice QA Systems?

Krunal Shah, Nitish Gupta, Dan Roth

Abstract

The recent success of machine learning systems on various QA datasets could be interpreted as a significant improvement in models' language understanding abilities. However, using various perturbations, multiple recent works have shown that good performance on a dataset might not indicate performance that correlates well with human's expectations from models that "understand" language. In this work we consider a top performing model on several Multiple Choice Question Answering (MCQA) datasets, and evaluate it against a set of expectations one might have from such a model, using a series of zero-information perturbations of the model's inputs. Our results show that the model clearly falls short of our expectations, and motivates a modified training approach that forces the model to better attend to the inputs. We show that the new training paradigm leads to a model that performs on par with the original model while better satisfying our expectations. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?Nishant Balepur, Abhilasha Ravichander, Rachel RudingerACL 2024 · 5 citations
  2. DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence UnderstandingXiaoyu Guo, Yuan-Fang Li, Gholamreza HaffariEMNLP 2023 · 1 citation

Builds on4

  1. Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
  2. Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
  3. QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations
  4. Good-Enough Compositional Data AugmentationJacob AndreasACL 2020 · 15 citations

Related papers

  1. Enhancing Multiple-choice Machine Reading Comprehension by Punishing Illogical InterpretationsYiming Ju, Yuanzhe Zhang, Zhixing Tian, Kang Liu et al.EMNLP 2021 · 8 citations
  2. Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice QuestionsSarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi et al.ICLR 2025
  3. LaSQuE: Improved Zero-Shot Classification from Explanations Through Quantifier Modeling and Curriculum LearningSayan Ghosh, Rakesh R. Menon, Shashank SrivastavaACL Findings 2023 · 2 citations
  4. Assessing the Benchmarking Capacity of Machine Reading Comprehension DatasetsSaku Sugawara, Pontus Stenetorp, Kentaro Inui, Akiko AizawaAAAI 2020 · 92 citations
  5. Leveraging Large Language Models for Multiple Choice Question AnsweringJoshua Robinson, David WingateICLR 2023 · 40 citations
Dusk over the sea between two cliffs drawn in fine vertical lines