Lune

ACL Findings2026

Interpretability from the Ground Up: Stakeholder-Centric Design of Automated Scoring in Educational Assessments

Yunsung Kim, Michael Hardy, Joseph Tey, Candace Thille, Christopher Piech

Abstract

AI-driven automated scoring systems offer scalable and efficient means of evaluating complex student-generated responses. Yet, despite increasing demand for transparency and interpretability, the field has yet to develop a widely accepted solution for interpretable automated scoring to be used in large-scale real-world assessments. This work takes a principled approach to address this challenge. We analyze the needs and potential benefits of interpretable automated scoring for various assessment stakeholder groups and develop four principles of interpretability -Faithfulness, Groundedness, Traceability, and Interchangeability (FGTI)targeted at those needs. To illustrate the feasibility of implementing these principles, we develop the ANALYTICSCORE framework 1 as a reference framework. When applied to the domain of text-based constructed-response scoring, ANALYTICSCORE outperforms many uninterpretable scoring methods in terms of scoring accuracy and is, on average, within 0.06 QWK of the uninterpretable SOTA across 10 items from the ASAP-SAS dataset. By comparing against human annotators conducting the same featurization task, we further demonstrate that the featurization behavior of ANALYTICSCORE aligns well with that of humans.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  2. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  3. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations
  4. Concept Bottleneck ModelsPang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann et al.ICML 2020 · 1,233 citations
  5. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations

Related papers

  1. From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality AssessmentIvo Bueno, Babette Bühler, Philipp Stark, Tim Fütterer et al.ACL Findings 2026
  2. PsyScore: A Psychometrically-Aware Framework for Trait-Adaptive Essay Scoring and ZPD-Scaffolded FeedbackWei Xia, Jin Wu, Haoran Shi, Xiangyu Wang et al.ACL Findings 2026 · 1 citation
  3. Measuring What Matters!! Assessing Therapeutic Principles in Mental-Health ConversationAbdullah Mazhar, Het Riteshkumar Shah, Aseem Srivastava, Smriti Joshi et al.ACL 2026
  4. Improving Domain Generalization for Prompt-Aware Essay Scoring via Disentangled Representation LearningZhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu et al.ACL 2023 · 16 citations
  5. Interpretable Knowledge Tracing via Response Influence-based Counterfactual ReasoningJiajun Cui, Minghe Yu, Bo Jiang, Aimin Zhou et al.ICDE 2024 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines