ACL Findings2022
Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks
Abstract
While fine-tuning pre-trained models for downstream classification is the conventional paradigm in NLP, often task-specific nuances may not get captured in the resultant models. Specifically, for tasks that take two inputs and require the output to be invariant of the order of the inputs, inconsistency is often observed in the predicted labels or confidence scores. We highlight this model shortcoming and apply a consistency loss function to alleviate inconsistency in symmetric classification. Our results show an improved consistency in predictions for three paraphrase detection datasets without a significant drop in the accuracy scores. We examine the classification performance of six datasets (both symmetric and non-symmetric) to showcase the strengths and limitations of our approach.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Logical Consistency of Large Language Models in Fact-Checking cites Improving Language Models' Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary
- Unveiling the Bias Impact on Symmetric Moral Consistency of Large Language Models cites Consistency Analysis of ChatGPT
- On Position Embeddings in BERT cites What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encoding
- Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models cites Consistency Analysis of ChatGPT
Cited by top-tier papers5
- Consistency Analysis of ChatGPTMyeongjun Jang, Thomas LukasiewiczEMNLP 2023 · 55 citations
- Improving Language Models' Meaning Understanding and Consistency by Learning Conceptual Roles from DictionaryMyeongjun Jang, Thomas LukasiewiczEMNLP 2023 · 1 citation
- Logical Consistency of Large Language Models in Fact-CheckingBishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat, Arijit KhanICLR 2025
- Unveiling the Bias Impact on Symmetric Moral Consistency of Large Language ModelsZiyi Zhou, Xinwei Guo, Jiashi Gao, Xiangyu Zhao et al.NeurIPS 2024
- Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language ModelsYinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi et al.ICML 2025
Builds on3
- On Position Embeddings in BERTBenyou Wang, Lifeng Shang, Christina Lioma, Xin Jiang et al.ICLR 2021 · 129 citations
- What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional EncodingYu-An Wang, Yun-Nung ChenEMNLP 2020 · 71 citations
- BLEURT: Learning Robust Metrics for Text GenerationThibault Sellam, Dipanjan Das, Ankur P. ParikhACL 2020 · 40 citations
Related papers
- Consistency Regularization for Cross-Lingual Fine-TuningBo Zheng, Li Dong, Shaohan Huang, Wenhui Wang et al.ACL 2021
- SaCo Loss: Sample-Wise Affinity Consistency for Vision-Language Pre-TrainingSitong Wu, Haoru Tan, Zhuotao Tian, Yukang Chen et al.CVPR 2024 · 5 citations
- Mitigating Forgetting in Adapting Pre-trained Language Models to Text Processing Tasks via Consistency AlignmentJianqi Gao, Hao Wu, Yiu-ming Cheung, Jian Cao et al.WWW 2025 · 4 citations
- Reconcile Prediction Consistency for Balanced Object DetectionKeyang Wang, Lei ZhangICCV 2021 · 36 citations
- Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text ClassificationVarsha Suresh, Desmond C. OngEMNLP 2021 · 71 citations
