EMNLP Findings2023
Ecologically Valid Explanations for Label Variation in NLI
Nan-Jiang Jiang, Chenhao Tan, Marie-Catherine de Marneffe
Abstract
Human label variation exists in many natural language processing (NLP) tasks, including natural language inference (NLI). To gain direct evidence of how NLI label variation arises, we build LIVENLI, an English dataset of 1,415 ecologically valid explanations (annotators explain the NLI labels they chose) for 122 MNLI (Williams et al., 2018) items (at least 10 explanations per item). The LIVENLI explanations confirm that people can systematically vary on their interpretation and highlight within-label variation: annotators sometimes choose the same label for different reasons. This suggests that explanations are crucial for navigating label interpretations in general. We few-shot prompt language models (LMs) to generate explanations but the results are inconsistent: the models sometimes produce valid and informative explanations, but they also generate implausible ones that do not support the label, highlighting directions for improvement.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Can language models learn from explanations in context? cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites VariErr NLI: Separating Annotation Error from Human Label Variation
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation cites LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference
- Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
- Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference cites VariErr NLI: Separating Annotation Error from Human Label Variation
- Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language Inference cites LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference
- The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
- VariErr NLI: Separating Annotation Error from Human Label Variation cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
- LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language Inference cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
Cited by top-tier papers5
- From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLPAdithya V. Ganesan, Vasudha Varadarajan, Oscar N. E. Kjell, Whitney Ringwald et al.ACL 2026 · 3 citations
- LiTEx: A Linguistic Taxonomy of Explanations for Understanding Within-Label Variation in Natural Language InferencePingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe et al.EMNLP 2025 · 1 citation
- Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label VariationBeiduo Chen, Yang Janet Liu, Anna Korhonen, Barbara PlankEMNLP 2025
- Label and Explanation Variation in LLM-Based Annotation: a Case Study in Natural Language InferenceArtur Kulmizev, Erika Lombart, Patrick Watrin, Marie-Catherine de MarneffeACL 2026
- VariErr NLI: Separating Annotation Error from Human Label VariationLeon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe, Barbara PlankACL 2024
Builds on6
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- The Unreliability of Explanations in Few-shot Prompting for Textual ReasoningXi Ye, Greg DurrettNeurIPS 2022 · 272 citations
- Can language models learn from explanations in context?Andrew K. Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory W. Mathewson et al.EMNLP Findings 2022 · 142 citations
- What Can We Learn from Collective Human Opinions on Natural Language Inference Data?Yixin Nie, Xiang Zhou, Mohit BansalEMNLP 2020 · 77 citations
- Distributed NLI: Learning to Predict Human Opinion Distributions for Language ReasoningXiang Zhou, Yixin Nie, Mohit BansalACL Findings 2022 · 13 citations
Related papers
- A Rose by Any Other Name: LLM-Generated Explanations Are Good Proxies for Human Explanations to Collect Label Distributions on NLIBeiduo Chen, Siyao Peng, Anna Korhonen, Barbara PlankACL Findings 2025 · 2 citations
- "Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko et al.EMNLP Findings 2024 · 2 citations
- Agree, Disagree, Explain: Decomposing Human Label Variation in NLI through the Lens of ExplanationsPingjun Hong, Beiduo Chen, Siyao Peng, Marie-Catherine de Marneffe et al.ACL Findings 2026
- Few-Shot Out-of-Domain Transfer Learning of Natural Language Explanations in a Label-Abundant SetupYordan Yordanov, Vid Kocijan, Thomas Lukasiewicz, Oana-Maria CamburuEMNLP Findings 2022 · 8 citations
- LIREx: Augmenting Language Inference with Relevant ExplanationsXinyan Zhao, V. G. Vinod VydiswaranAAAI 2021 · 41 citations
