Lune

ACL Findings2022

Capture Human Disagreement Distributions by Calibrated Networks for Natural Language Inference

Yuxia Wang, Minghan Wang, Yimeng Chen, Shimin Tao, Jiaxin Guo, Chang Su, Min Zhang, Hao Yang

Abstract

Natural Language Inference (NLI) datasets contain examples with highly ambiguous labels due to its subjectivity. Several recent efforts have been made to acknowledge and embrace the existence of ambiguity, and explore how to capture the human disagreement distribution. In contrast with directly learning from gold ambiguity labels, relying on special resource, we argue that the model has naturally captured the human ambiguity distribution as long as it's calibrated, i.e. the predictive probability can reflect the true correctness likelihood. Our experiments show that when model is well-calibrated, either by label smoothing or temperature scaling, it can obtain competitive performance as prior work, on both divergence scores between predictive probability and the true human opinion distribution, and the accuracy. This reveals the overhead of collecting gold ambiguity labels can be cut, by broadly solving how to calibrate the NLI network.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Deep Model Compression Also Helps Models Capture Ambiguity cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
  • Can Large Language Models Capture Dissenting Human Voices? cites What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Can Large Language Models Capture Dissenting Human Voices?Noah Lee, Na An, James ThorneEMNLP 2023 · 8 citations
  2. Deep Model Compression Also Helps Models Capture AmbiguityHancheol Park, Jong C. ParkACL 2023
  3. A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference CorpusEduard Gabriel Poesina, Cornelia Caragea, Radu Tudor IonescuACL 2024

Builds on2

  1. Does label smoothing mitigate label noise?Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, Sanjiv KumarICML 2020 · 411 citations
  2. What Can We Learn from Collective Human Opinions on Natural Language Inference Data?Yixin Nie, Xiang Zhou, Mohit BansalEMNLP 2020 · 77 citations

Related papers

  1. Distributed NLI: Learning to Predict Human Opinion Distributions for Language ReasoningXiang Zhou, Yixin Nie, Mohit BansalACL Findings 2022 · 13 citations
  2. We're Afraid Language Models Aren't Modeling AmbiguityAlisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr et al.EMNLP 2023 · 35 citations
  3. On the Inference Calibration of Neural Machine TranslationShuo Wang, Zhaopeng Tu, Shuming Shi, Yang LiuACL 2020 · 66 citations
  4. You Are What You Annotate: Towards Better Models through Annotator RepresentationsNaihao Deng, Xinliang Frederick Zhang, Siyang Liu, Winston Wu et al.EMNLP Findings 2023 · 5 citations
  5. DiCaP: Distribution-Calibrated Pseudo-labeling for Semi-Supervised Multi-Label LearningBo Han, Zhuoming Li, Xiaoyu Wang, Yaxin Hou et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines