ACL Findings2026
Take Out Your Calculators: Estimating the Real Difficulty of Question Items with LLM Student Simulations
Christabel Acquaye, Yi Ting Huang, Marine Carpuat, Rachel Rudinger
Abstract
Standardized math assessments require expensive human pilot studies to establish the difficulty of test items. We investigate the predictive value of open-source large language models (LLMs) for evaluating the difficulty of multiple-choice math questions for real-world students. We show that, while LLMs are poor direct judges of problem difficulty, simulationbased approaches with LLMs yield promising results under the right conditions. Under the proposed approach, we simulate a "classroom" of 4th, 8th, or 12th grade students by prompting the LLM to role-play students of varying proficiency levels. We use the outcomes of these simulations to fit Item Response Theory (IRT) models, comparing learned difficulty parameters for items to their real-world difficulties, as determined by item-level statistics furnished by the National Assessment of Educational Progress (NAEP). We observe correlations as high as 0.75, 0.76, and 0.82 for grades 4, 8, and 12, respectively on the item-level correctness rates. In our simulations, we experiment on math MCQs with different "classroom sizes," showing tradeoffs between computation size and accuracy. We find that roleplays with diverse named students improves predictions (compared to student ids), and stratifying names across gender and race further improves predictions. Our results show that LLMs with relatively weaker mathematical abilities (Gemma) actually yield better real-world difficulty predictions than mathematically stronger models (Llama and Qwen), further underscoring the suitability of these models for the task.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on9
- Can Large Language Model Agents Simulate Human Trust Behavior?Chengxing Xie, Canyu Chen, Feiran Jia, Ziyu Ye et al.NeurIPS 2024 · 183 citations
- Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language ModelsRobert Wolfe, Aylin CaliskanEMNLP 2021 · 27 citations
- When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT DifficultyFernando Martínez-Plumed, David Castellano Falcón, Carlos Monserrat Aranda, José Hernández-OralloAAAI 2022 · 18 citations
- Using LLMs to simulate students' responses to exam questionsLuca Benedetto, Giovanni Aradelli, Antonia Donvito, Alberto Lucchetti et al.EMNLP Findings 2024 · 14 citations
- Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral SimulationSonglin Xu, Hao-Ning Wen, Hongyi Pan, Dallas Dominguez et al.CHI 2025 · 13 citations
Related papers
- Graph-Based Alternatives to LLMs for Human SimulationJoseph Suh, Suhong Moon, Serina ChangACL 2026 · 2 citations
- SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty PredictionAlexander Scarlatos, Nigel Fernandez, Christopher Ormerod, Susan Lottridge et al.EMNLP 2025
- VisNumBench: Evaluating Number Sense of Multimodal Large Language ModelsTengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong MingICCV 2025 · 1 citation
- Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem SolvingYuxuan Zhou, Xien Liu, Chenwei Yan, Chen Ning et al.ICML 2025
- Simulated Students in Tutoring Dialogues: Substance or Illusion?Alexander Scarlatos, Jaewook Lee, Simon Woodhead, Andrew LanACL 2026 · 6 citations
