Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

Estimating question difficulty is a critical component in evaluating and improving large language models (LLMs) for question answering (QA). Existing approaches often rely on readability formulas, retrieval-based signals, or popularity statistics, which may not fully capture the reasoning challenges posed to modern LLMs. In this paper, we introduce Q-DAPS (Question Difficulty based on Answer Plausibility Scores) method, a novel approach that estimates question difficulty by computing the entropy of plausibility scores over candidate answers. We systematically evaluate Q-DAPS across four prominent QA datasets-TriviaQA, NQ, MuSiQue, and QASC-demonstrating that it consistently outperforms baselines. Moreover, Q-DAPS shows strong robustness across hyperparameter variations and question types. Extensive ablation studies further show that Q-DAPS remains robust across different plausibility estimation paradigms, model sizes, and realistic settings. Human evaluations further confirm strong alignment between Q-DAPS 's difficulty estimates and human judgments of question difficulty. Overall, Q-DAPS provides an interpretable, scalable, and bias-resilient approach to question difficulty estimation in modern QA systems. https://github.com/DataScienceUIBK/ Q-DAPS 1 Introduction Questions are a fundamental means by which users express their information needs in Information Retrieval (IR) and Natural Language Processing (NLP) systems. They span a wide range of types-factoid, definition, and yes/no (Pandya and Bhatt, 2021)-and can also be classified by difficulty (Benedetto et al., 2023), such as easy, *