Lune

ACL Findings2023

SamToNe: Improving Contrastive Loss for Dual Encoder Retrieval Models with Same Tower Negatives

Fedor Moiseev, Gustavo Hernández Abrego, Péter Dornbach, Imed Zitouni, Enrique Alfonseca, Zhe Dong

Abstract

Dual encoders have been used for retrieval tasks and representation learning with good results. A standard way to train dual encoders is using a contrastive loss with in-batch negatives. In this work, we propose an improved contrastive learning objective by adding queries or documents from the same encoder towers to the negatives, for which we name it as "contrastive loss with SAMe TOwer NEgatives" (SamToNe). By evaluating on question answering retrieval benchmarks from MS MARCO and MultiReQA, and heterogenous zero-shot information retrieval benchmarks (BEIR), we demonstrate that SamToNe can effectively improve the retrieval quality for both symmetric and asymmetric dual encoders. By directly probing the embedding spaces of the two encoding towers via the t-SNE algorithm (van der Maaten and Hinton, 2008), we observe that SamToNe ensures the alignment between the embedding spaces from the two encoder towers. Based on the analysis of the embedding distance distributions of the top-1 retrieved results, we further explain the efficacy of the method from the perspective of regularisation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Large Dual Encoders Are Generalizable Retrievers cites Dense Passage Retrieval for Open-Domain Question Answering
  • Large Dual Encoders Are Generalizable Retrievers cites Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
  • HOBIT: Hardness Optimized Batch Sampling for InfoNCE Training cites Dense Passage Retrieval for Open-Domain Question Answering
  • HOBIT: Hardness Optimized Batch Sampling for InfoNCE Training cites PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval
  • PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval cites Dense Passage Retrieval for Open-Domain Question Answering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. HOBIT: Hardness Optimized Batch Sampling for InfoNCE TrainingHimanshu Dutta, Lokesh Nagalapatti, Yashoteja PrabhuICML 2026

Builds on4

  1. Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text ModelsJianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma et al.ACL Findings 2022 · 358 citations
  2. Large Dual Encoders Are Generalizable RetrieversJianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai et al.EMNLP 2022 · 145 citations
  3. Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
  4. PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage RetrievalRuiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu et al.ACL Findings 2021 · 59 citations

Related papers

  1. Contextual Document EmbeddingsJohn Xavier Morris, Alexander M. RushICLR 2025
  2. LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text RetrievalCanwen Xu, Daya Guo, Nan Duan, Julian J. McAuleyACL Findings 2022 · 26 citations
  3. Towards Robust Ranker for Text RetrievalYucheng Zhou, Tao Shen, Xiubo Geng, Chongyang Tao et al.ACL Findings 2023 · 15 citations
  4. CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video RepresentationsMohammadreza Zolfaghari, Yi Zhu, Peter V. Gehler, Thomas BroxICCV 2021 · 160 citations
  5. Adversarial Retriever-Ranker for Dense Text RetrievalHang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv et al.ICLR 2022 · 137 citations
Dusk over the sea between two cliffs drawn in fine vertical lines