ACL Findings2025
A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLI
Beiduo Chen, Siyao Peng, Anna Korhonen, Barbara Plank
Abstract
Disagreement in human labeling is ubiquitous, and can be captured in human judgment distributions (HJDs). Recent research has shown that explanations provide valuable information for understanding human label variation (HLV) and large language models (LLMs) can approximate HJD from a few human-provided labelexplanation pairs. However, collecting explanations for every label is still time-consuming. This paper examines whether LLMs can be used to replace humans in generating explanations for approximating HJD. Specifically, we use LLMs as annotators to generate model explanations for a few given human labels. We test ways to obtain and combine these labelexplanations with the goal to approximate human judgment distributions. We further compare the resulting human with model-generated explanations, and test automatic and human explanation selection. Our experiments show that LLM explanations are promising for NLI: to estimate HJDs, generated explanations yield comparable results to human's when provided with human labels. Importantly, our results generalize from datasets with human explanations to i) datasets where they are not available and ii) challenging out-of-distribution test sets. Distributions Dist. Comparison BERT Fine-Tuning Comparison (dev/test) RoBERTa Fine-Tuning Comparison (dev/test) Global Metric
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites Measuring Massive Multitask Language Understanding
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites Whose Opinions Do Language Models Reflect?
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference
- Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference cites LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- Whose Opinions Do Language Models Reflect? cites Measuring Massive Multitask Language Understanding
- Large Language Models Are Not Robust Multiple Choice Selectors cites Measuring Massive Multitask Language Understanding
- LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference cites Adversarial NLI: A New Benchmark for Natural Language Understanding
Cited by top-tier papers4
- LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language InferencePingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe et al.EMNLP 2025 · 1 citation
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label VariationBeiduo Chen, Yang Janet Liu, Anna Korhonen, Barbara PlankEMNLP 2025
- Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language InferenceArtur Kulmizev, Erika Lombart, Patrick Watrin, Marie-Catherine de MarneffeACL 2026
- Exploring Selective Avoidance for Online User Behavior Analysis: A Forest of Thought ExplanationXiaohua Wu, Lin Li, Kaize Shi, Xiaohui Tao et al.AAAI 2026
Builds on13
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Whose Opinions Do Language Models Reflect?Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee et al.ICML 2023 · 764 citations
- Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
- Large Language Models Are Not Robust Multiple Choice SelectorsChujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou et al.ICLR 2024 · 424 citations
Related papers
- "Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko et al.EMNLP Findings 2024 · 2 citations
- Human-LLM Collaborative Annotation Through Effective Verification of LLM LabelsXinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra et al.CHI 2024 · 127 citations
- Ecologically Valid Explanations for Label Variation in NLINan-Jiang Jiang, Chenhao Tan, Marie-Catherine de MarneffeEMNLP Findings 2023
- EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLILongfei Zuo, Barbara Plank, Siyao PengACL Findings 2026
- Can Large Language Models Capture Dissenting Human Voices?Noah Lee, Na An, James ThorneEMNLP 2023 · 8 citations
