ACL Findings2021
Contrastive Fine-tuning Improves Robustness for Neural Rankers
Xiaofei Ma, Cícero Nogueira dos Santos, Andrew O. Arnold
Abstract
The performance of state-of-the-art neural rankers can deteriorate substantially when exposed to noisy inputs or applied to a new domain. In this paper, we present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. Specifically, a contrastive loss that compares data points in the representation space is combined with the standard ranking loss during fine-tuning. We use relevance labels to denote similar/dissimilar pairs, which allows the model to learn the underlying matching semantics across different query-document pairs and leads to improved robustness. In experiments with four passage ranking datasets, the proposed contrastive fine-tuning method obtains improvements on robustness to query reformulations, noise perturbations, and zeroshot transfer for both BERT and BART-based rankers. Additionally, our experiments show that contrastive fine-tuning outperforms data augmentation for robustifying neural rankers.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers3
- A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular SpaceYuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu et al.ACL 2022 · 94 citations
- Successive Prompting for Decomposing Complex QuestionsDheeru Dua, Shivanshu Gupta, Sameer Singh, Matt GardnerEMNLP 2022 · 38 citations
- UNICS: Multilingual Code Search via Unified Pseudocode and Contrastive Transfer LearningYe Fan, Jidong Ge, Chuanyi Li, LiGuo Huang et al.FSE 2026
Builds on9
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text RetrievalLee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang et al.ICLR 2021 · 1,547 citations
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and EntailmentDi Jin, Zhijing Jin, Joey Tianyi Zhou, Peter SzolovitsAAAI 2020 · 1,333 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Learning The Difference That Makes A Difference With Counterfactually-Augmented DataDivyansh Kaushik, Eduard H. Hovy, Zachary Chase LiptonICLR 2020 · 625 citations
Related papers
- Towards Robust Ranker for Text RetrievalYucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao et al.ACL Findings 2023 · 15 citations
- Improved Text Classification via Contrastive Adversarial TrainingLin Pan, Chung-Wei Hang, Avirup Sil, Saloni PotdarAAAI 2022 · 115 citations
- COCO-DR: Combating the Distribution Shift in Zero-Shot Dense Retrieval with Contrastive and Distributionally Robust LearningYue Yu, Chenyan Xiong, Si Sun, Chao Zhang et al.EMNLP 2022 · 21 citations
- Self-Guided Contrastive Learning for BERT Sentence RepresentationsTaeuk Kim, Kang Min Yoo, Sang-goo LeeACL 2021
- Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-TuningEuna Jung, Jaekeol Choi, Wonjong RheeWWW 2022 · 15 citations
