BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks

Multiple-choice question answering (MCQA) is standard in NLP, but benchmarks lack rigorous quality control. We present BenchMarker, an education-inspired toolkit using LLM judges to flag three common MCQ flaws: 1) contamination-items appearing exactly online; 2) shortcuts-cues in the choices that enable guessing; and 3) writing errors-structural/grammatical issues based on a 19-rule education rubric. We validate BenchMarker with human annotations, then run the tool to audit 12 benchmarks, revealing: 1) flaws persist in MCQA benchmarks, especially automatically-made and crowdsourced data-we detect 47% of TruthfulQA appears online and 100% of HellaSwag violates multiple writing rules; 2) contaminated MCQs tend to inflate accuracy, while writing errors tend to lower it and change rankings beyond random; and 3) prior benchmark repairs address their targeted issues (i.e., lowering accuracy with LLMwritten distractors), but inadvertently add new flaws (i.e. implausible distractors, many correct answers). Overall, flaws in MCQs degrade NLP evaluation, but education research offers a path forward. We release BenchMarker to bridge the fields and improve MCQA benchmark design. 1 Multiple-Choice Question j MCQ j … MCQ j … MCQ j … The appropriate place… www.quizlet.com Which can you drink? Inferred Question Rule 7: Is grammar consistent? 19-Rule Rubric from Education 1 (no question match) 0 (matches question) 1 (violates rule) 0 (no rule violation) Guess w/out question 1 (exists online) 0 (not online) + Contamination ( §3.1) Shortcuts ( §3.2) Writing Errors ( §3.3) Web search Report Card Contamination: 0.21 > Item j was found online Shortcuts: 0.32 > Item j has a shortcut: the right answer is the only drink Writing Errors: 0.45 > Item j R7: "This item" has a grammar mismatch -"plates" MCQA Benchmark BenchMarker: Highlighting Flaws Question: The appropriate place to put this item is a recycling bin Choices: (A) used motor oil (B) used soda can (C) used styrofoam plates (D) left over medicine Answer: (B)