ACL Findings2022
Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding
Rui Cao, Yihao Wang, Yuxin Liang, Ling Gao, Jie Zheng, Jie Ren, Zheng Wang
Abstract
Contrastive learning is emerging as a powerful technique for extracting knowledge from unlabeled data. This technique requires a balanced mixture of two ingredients: positive (similar) and negative (dissimilar) samples. This is typically achieved by maintaining a queue of negative samples during training. Prior works in the area typically uses a fixed-length negative sample queue, but how the negative sample size affects the model performance remains unclear. The opaque impact of the number of negative samples on performance when employing contrastive learning aroused our in-depth exploration. This paper presents a momentum contrastive learning model with negative sample queue for sentence embedding, namely MoCoSE. We add the prediction layer to the online branch to make the model asymmetric and together with EMA update mechanism of the target branch to prevent the model from collapsing. We define a maximum traceable distance metric, through which we learn to what extent the text contrastive learning benefits from the historical information of negative samples. Our experiments find that the best results are obtained when the maximum traceable distance is at a certain range, demonstrating that there is an optimal range of historical information for a negative sample queue. We evaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS) task and obtain an average Spearman's correlation of 77.27%. Source code is available here.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere cites A Simple Framework for Contrastive Learning of Visual Representations
- Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations cites A Simple Framework for Contrastive Learning of Visual Representations
- Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- HOBIT: Hardness Optimized Batch Sampling for InfoNCE Training cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Composition-contrastive Learning for Sentence Embeddings cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Composition-contrastive Learning for Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
- Composition-contrastive Learning for Sentence Embeddings cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites A Simple Framework for Contrastive Learning of Visual Representations
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction cites A Simple Framework for Contrastive Learning of Visual Representations
- Barlow Twins: Self-Supervised Learning via Redundancy Reduction cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
Cited by top-tier papers5
- Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense RelationsWanqiu Long, Bonnie WebberEMNLP 2022 · 15 citations
- Composition-contrastive Learning for Sentence EmbeddingsSachin Chanchani, Ruihong HuangACL 2023 · 11 citations
- miCSE: Mutual Information Contrastive Learning for Low-shot Sentence EmbeddingsTassilo Klein, Moin NabiACL 2023 · 10 citations
- Modeling Sequential Sentence Relation to Improve Cross-lingual Dense RetrievalShunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang et al.ICLR 2023 · 1 citation
- HOBIT: Hardness Optimized Batch Sampling for InfoNCE TrainingHimanshu Dutta, Lokesh Nagalapatti, Yashoteja PrabhuICML 2026
Builds on16
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Barlow Twins: Self-Supervised Learning via Redundancy ReductionJure Zbontar, Li Jing, Ishan Misra, Yann LeCun et al.ICML 2021 · 2,942 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the HypersphereTongzhou Wang, Phillip IsolaICML 2020 · 2,360 citations
Related papers
- InfoCSE: Information-aggregated Contrastive Learning of Sentence EmbeddingsXing Wu, Chaochen Gao, Zijia Lin, Jizhong Han et al.EMNLP Findings 2022 · 24 citations
- A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence EmbeddingsHaochen Tan, Wei Shao, Han Wu, Ke Yang et al.ACL Findings 2022 · 16 citations
- TNCSE: Tensor Norm Constraints for Unsupervised Contrastive Learning of Sentence EmbeddingsTianyu Zong, Bingkang Shi, Hongzhu Yi, Jungang XuAAAI 2025 · 2 citations
- Instance Smoothed Contrastive Learning for Unsupervised Sentence EmbeddingHongliang He, Junlei Zhang, Zhenzhong Lan, Yue ZhangAAAI 2023 · 10 citations
- Unsupervised Sentence Representation via Contrastive Learning with Mixing NegativesYanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu et al.AAAI 2022 · 71 citations
