Lune

ACL Findings2022

Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning

Xiang Zhou, Yixin Nie, Mohit Bansal

Abstract

We introduce distributed NLI, a new NLU task with a goal to predict the distribution of human judgements for natural language inference. We show that by applying additional distribution estimation methods, namely, Monte Carlo (MC) Dropout, Deep Ensemble, Re-Calibration, and Distribution Distillation, models can capture human judgement distribution more effectively than the softmax baseline. We show that MC Dropout is able to achieve decent performance without any distribution annotations while Re-Calibration can give further improvements with extra distribution annotations, suggesting the value of multiple annotations for one example in modeling the distribution of human judgements. Despite these improvements, the best results are still far below the estimated human upper-bound, indicating that predicting the distribution of human judgements is still an open, challenging problem with a large room for improvements. We showcase the common errors for MC Dropout and Re-Calibration. Finally, we give guidelines on the usage of these methods with different levels of data availability and encourage future work on modeling the human opinion distribution for language reasoning. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Adversarial NLI: A New Benchmark for Natural Language Understanding cites Abductive Commonsense Reasoning
  • The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality cites Human Uncertainty Makes Classification More Robust
  • Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
  • Learning with Different Amounts of Annotation: From Zero to Many Labels cites Abductive Commonsense Reasoning
  • Learning with Different Amounts of Annotation: From Zero to Many Labels cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
  • What Can We Learn from Collective Human Opinions on Natural Language Inference Data? cites Adversarial NLI: A New Benchmark for Natural Language Understanding
  • What Can We Learn from Collective Human Opinions on Natural Language Inference Data? cites Abductive Commonsense Reasoning
  • We're Afraid Language Models Aren't Modeling Ambiguity cites Learning with Different Amounts of Annotation: From Zero to Many Labels
  • Practical estimation of the optimal classification error with soft labels and calibration cites Human Uncertainty Makes Classification More Robust
  • Practical estimation of the optimal classification error with soft labels and calibration cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
  • Can Large Language Models Capture Dissenting Human Voices? cites Adversarial NLI: A New Benchmark for Natural Language Understanding
  • Can Large Language Models Capture Dissenting Human Voices? cites The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With Reality
  • Can Large Language Models Capture Dissenting Human Voices? cites Learning with Different Amounts of Annotation: From Zero to Many Labels
  • Can Large Language Models Capture Dissenting Human Voices? cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
  • Can Large Language Models Capture Dissenting Human Voices? cites We're Afraid Language Models Aren't Modeling Ambiguity
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. We're Afraid Language Models Aren't Modeling AmbiguityAlisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr et al.EMNLP 2023 · 35 citations
  2. Learning with Different Amounts of Annotation: From Zero to Many LabelsShujian Zhang, Chengyue Gong, Eunsol ChoiEMNLP 2021 · 21 citations
  3. Interface Design for Crowdsourcing Hierarchical Multi-Label Text AnnotationsRickard Stureborg, Bhuwan Dhingra, Jun YangCHI 2023 · 9 citations
  4. Can Large Language Models Capture Dissenting Human Voices?Noah Lee, Na An, James ThorneEMNLP 2023 · 8 citations
  5. Practical estimation of the optimal classification error with soft labels and calibrationRyota Ushio, Takashi Ishida, Masashi SugiyamaICLR 2026 · 6 citations

Builds on7

  1. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
  2. Abductive Commonsense ReasoningChandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi et al.ICLR 2020 · 521 citations
  3. Human Uncertainty Makes Classification More RobustJoshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, Olga RussakovskyICCV 2019 · 362 citations
  4. The Disagreement Deconvolution: Bringing Machine Learning Performance Metrics In Line With RealityMitchell L. Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto et al.CHI 2021 · 100 citations
  5. What Can We Learn from Collective Human Opinions on Natural Language Inference Data?Yixin Nie, Xiang Zhou, Mohit BansalEMNLP 2020 · 77 citations

Related papers

  1. Capture Human Disagreement Distributions by Calibrated Networks for Natural Language InferenceYuxia Wang, Minghan Wang, Yimeng Chen, Shimin Tao et al.ACL Findings 2022 · 5 citations
  2. Escaping the Mode: Multi-Answer Reinforcement Learning in LMsIsha Puri, Mehul Damani, Idan Shenfeld, Marzyeh Ghassemi et al.ICML 2026
  3. Disagreement Matters: Preserving Label Diversity by Jointly Modeling Item and Annotator Label Distributions with DisCoTharindu Cyril Weerasooriya, Alexander Ororbia, Raj Bhensadadia, Ashiqur R. KhudaBukhsh et al.ACL Findings 2023 · 4 citations
  4. "Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?Beiduo Chen, Xinpeng Wang, Siyao Peng, Robert Litschko et al.EMNLP Findings 2024 · 2 citations
  5. OpenEstimate: Evaluating LLMs on Reasoning Under Uncertainty with Real-World DataAlana Renda, Jillian Ross, Jacob AndreasICLR 2026 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines