Lune

ACL Findings2022

Mitigating the Inconsistency Between Word Saliency and Model Confidence with Pathological Contrastive Training

Pengwei Zhan, Yang Wu, Shaolei Zhou, Yunjian Zhang, Liming Wang

Abstract

Neural networks are widely used in various NLP tasks for their remarkable performance. However, the complexity makes them difficult to interpret, i.e., they are not guaranteed right for the right reason. Besides the complexity, we reveal that the model pathology -the inconsistency between word saliency and model confidence, further hurts the interpretability. We show that the pathological inconsistency is caused by the representation collapse issue, which means that the representation of the sentences with tokens in different saliency reduced is somehow collapsed, and thus the important words cannot be distinguished from unimportant words in terms of model confidence changing. In this paper, to mitigate the pathology and obtain more interpretable models, we propose Pathological Contrastive Training (PCT) framework, which adopts contrastive learning and saliency-based samples augmentation to calibrate the sentences representation. Combined with qualitative analysis, we also conduct extensive quantitative experiments and measure the interpretability with eight reasonable metrics. Experiments show that our method can mitigate the model pathology and generate more interpretable models while keeping the model performance. Ablation study also shows the effectiveness.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Unveiling the Lexical Sensitivity of LLMs: Combinatorial Optimization for Prompt Enhancement cites Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model
  • Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model cites A Simple Framework for Contrastive Learning of Visual Representations
  • Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model cites ERASER: A Benchmark to Evaluate Rationalized NLP Models
  • Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model cites DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
  • Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language Model cites Big Self-Supervised Models are Strong Semi-Supervised Learners
  • A Simple Framework for Contrastive Learning of Visual Representations cites Local Aggregation for Unsupervised Learning of Visual Embeddings
  • Big Self-Supervised Models are Strong Semi-Supervised Learners cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Unveiling the Lexical Sensitivity of LLMs: Combinatorial Optimization for Prompt EnhancementPengwei Zhan, Zhen Xu, Qian Tan, Jie Song et al.EMNLP 2024 · 9 citations
  2. Contrastive Learning with Adversarial Examples for Alleviating Pathology of Language ModelPengwei Zhan, Jing Yang, Xiao Huang, Chunlei Jing et al.ACL 2023 · 1 citation

Builds on6

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Big Self-Supervised Models are Strong Semi-Supervised LearnersTing Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi et al.NeurIPS 2020 · 2,611 citations
  3. Local Aggregation for Unsupervised Learning of Visual EmbeddingsChengxu Zhuang, Alex Lin Zhai, Daniel YaminsICCV 2019 · 462 citations
  4. ERASER: A Benchmark to Evaluate Rationalized NLP ModelsJay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric P. Lehman et al.ACL 2020 · 36 citations
  5. DeCLUTR: Deep Contrastive Learning for Unsupervised Textual RepresentationsJohn M. Giorgi, Osvald Nitski, Bo Wang, Gary D. BaderACL 2021

Related papers

  1. Counterfactual-based Saliency Map: Towards Visual Contrastive Explanations for Neural NetworksXue Wang, Zhibo Wang, Haiqin Weng, Hengchang Guo et al.ICCV 2023 · 15 citations
  2. Explanation Consistency Training: Facilitating Consistency-Based Semi-Supervised Learning with InterpretabilityTao Han, Wei-Wei Tu, Yufeng LiAAAI 2021 · 25 citations
  3. Improving Deep Learning Interpretability by Saliency Guided TrainingAya Abdelsalam Ismail, Héctor Corrada Bravo, Soheil FeiziNeurIPS 2021 · 121 citations
  4. Graph Segmentation and Contrastive Enhanced Explainer for Graph Neural NetworksZhiqiang Wang, Jiayu Guo, Jianqing Liang, Jiye Liang et al.AAAI 2025 · 4 citations
  5. Interpreting Language Models with Contrastive ExplanationsKayo Yin, Graham NeubigEMNLP 2022 · 32 citations
Dusk over the sea between two cliffs drawn in fine vertical lines