Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks
Dongjun Kim, Gyuho Shim, Yongchan Chun, Minhyuk Kim, Chanjun Park, Heuiseok Lim
Abstract
Large Language Models are commonly judged by their scores on standard benchmarks, yet such scores often overstate real capability since they mask the mix of skills a task actually demands. For example, ARC is assumed to test reasoning, while HellaSwag is designed to evaluate commonsense. However, we lack a systematic way to verify if these benchmarks actually measure these labels. We introduce BENCHMARK PROFILING, a diagnostic framework that decomposes benchmark performance into ten cognitively grounded abilities. The method combines gradient-based importance scoring with targeted parameter ablation to compute an Ability Impact Score (AIS) that quantifies how much each ability contributes to a model's success on a given benchmark. Profiling three instruction-tuned models across ten widely used benchmarks yields four key findings: (i) most benchmarks draw on several abilities rather than one, (ii) datasets with similar labels rely on distinct ability mixtures, (iii) code-generation benchmarks reward broad, multi-skill improvement and thus show only modest gains from narrow domain-specific fine-tuning, and (iv) abilities irrelevant to the task could negatively affect performance. BENCHMARK PROFILING therefore explains why performance gains do not always translate into user-perceived competence and offers a transparent tool for benchmark audit and model interpretability. The code is available on https://github.com/ junkim100/Benchmark-Profiling
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- Unveiling Linguistic Regions in Large Language Models cites Measuring Massive Multitask Language Understanding
- Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysis cites A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis
Builds on5
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation AnalysisAlessandro Stolfo, Yonatan Belinkov, Mrinmaya SachanEMNLP 2023 · 11 citations
- Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron AnalysisZeping Yu, Sophia AnaniadouEMNLP 2024 · 1 citation
- Unveiling Linguistic Regions in Large Language ModelsZhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui et al.ACL 2024
Related papers
- From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate LLMs?Geng Zhang, Yizhou Ying, Sihang Jiang, Jiaqing Liang et al.EMNLP Findings 2025
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static BenchmarksPeiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng et al.ICML 2026 · 23 citations
- Predicting Fine-Tuning Performance with ProbingZining Zhu, Soroosh Shahtalebi, Frank RudziczEMNLP 2022 · 6 citations
- CDT: A Comprehensive Capability Framework for Large Language Models Across Cognition, Domain, and TaskHaosi Mo, Xinyu Ma, Xuebo Liu, Derek F. Wong et al.EMNLP Findings 2025
- ALERT: Adapt Language Models to Reasoning TasksPing Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissi et al.ACL 2023 · 4 citations
