EMNLP Findings2024
Plausibly Problematic Questions in Multiple-Choice Benchmarks for Commonsense Reasoning
Shramay Palta, Nishant Balepur, Peter Rankel, Sarah Wiegreffe, Marine Carpuat, Rachel Rudinger
Abstract
Questions involving commonsense reasoning about everyday situations often admit many possible or plausible answers.In contrast, multiple-choice question (MCQ) benchmarks for commonsense reasoning require a hard selection of a single correct answer, which, in principle, should represent the most plausible answer choice.On 250 MCQ items sampled from two commonsense reasoning benchmarks, we collect 5, 000 independent plausibility judgments on answer choices.We find that for over 20% of the sampled MCQS, the answer choice rated most plausible does not match the benchmark gold answers; upon manual inspection, we confirm that this subset exhibits higher rates of problems like ambiguity or semantic mismatch between question and answer choices.Experiments with LLMS reveal low accuracy and high variation in performance on the subset, suggesting our plausibility criterion may be helpful in identifying more reliable benchmark items for commonsense evaluation. 1AnswerA: relieved 👱: 5, 2, 5, 5, 4 (4.2) AnswerB: accomplished 👱: 4, 2, 5, 2, 5 (3.6)AnswerC: proud 👱: 4, 5, 5, 5, 5 (4.8)Context: Ash redeemed themselves after retaking the test they failed.Question: How will Ash feel as a result?
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above cites Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US
- Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the US cites PIQA: Reasoning about Physical Commonsense in Natural Language
- Thinking Like a Skeptic: Defeasible Inference in Natural Language cites Abductive Commonsense Reasoning
- Social Story Frames: Contextual Reasoning about Narrative Intent and Reception cites Abductive Commonsense Reasoning
- Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations cites Abductive Commonsense Reasoning
Cited by top-tier papers6
- Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the USChristabel Acquaye, Haozhe An, Rachel RudingerEMNLP 2024 · 2 citations
- Social Story Frames: Contextual Reasoning about Narrative Intent and ReceptionJoel Mire, Maria Antoniak, Steven R. Wilson, Zexin Ma et al.ACL 2026 · 2 citations
- BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice BenchmarksNishant Balepur, Bhavya Rajasekaran, Hyunjin Jane Oh, Michael Xie et al.ACL 2026 · 1 citation
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label VariationBeiduo Chen, Yang Janet Liu, Anna Korhonen, Barbara PlankEMNLP 2025
- Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the AboveNishant Balepur, Rachel Rudinger, Jordan Lee Boyd-GraberACL 2025
Builds on14
- PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
- Abductive Commonsense ReasoningChandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi et al.ICLR 2020 · 521 citations
- Maieutic Prompting: Logically Consistent Reasoning with Recursive ExplanationsJaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman et al.EMNLP 2022 · 72 citations
- Thinking Like a Skeptic: Defeasible Inference in Natural LanguageRachel Rudinger, Vered Shwartz, Jena D. Hwang, Chandra Bhagavatula et al.EMNLP Findings 2020 · 63 citations
- Rissanen Data Analysis: Examining Dataset Characteristics via Description LengthEthan Perez, Douwe Kiela, Kyunghyun ChoICML 2021 · 25 citations
Related papers
- From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment ItemsMelissa Roemmele, Andrew S. GordonEMNLP Findings 2024 · 12 citations
- Arguments that Alter Minds: LLM Rationales Sway Human (and LLM) Notions of PlausibilityShramay Palta, Peter Rankel, Sarah Wiegreffe, Rachel RudingerACL 2026
- ARC 'Challenge' Is Not That ChallengingLukasz BorchmannACL Findings 2025
- COM2SENSE: A Commonsense Reasoning Benchmark with Complementary SentencesShikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi et al.ACL Findings 2021 · 30 citations
- ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question AnsweringFrancesco Maria Molfese, Luca Moroni, Ciro Porcaro, Simone Conia et al.ACL 2026 · 1 citation
