ACL Findings2023
Clustering-Aware Negative Sampling for Unsupervised Sentence Representation
Jinghao Deng, Fanqi Wan, Tao Yang, Xiaojun Quan, Rui Wang
Abstract
Contrastive learning has been widely studied in sentence representation learning. However, earlier works mainly focus on the construction of positive examples, while in-batch samples are often simply treated as negative examples. This approach overlooks the importance of selecting appropriate negative examples, potentially leading to a scarcity of hard negatives and the inclusion of false negatives. To address these issues, we propose ClusterNS (Clusteringaware Negative Sampling), a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified Kmeans clustering algorithm to supply hard negatives and recognize in-batch false negatives during training, aiming to solve the two issues in one unified framework. Experiments on semantic textual similarity (STS) tasks demonstrate that our proposed ClusterNS compares favorably with baselines in unsupervised sentence representation learning. Our code has been made publicly available. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Hard Negative Mixing for Contrastive Learning cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Hard Negative Mixing for Contrastive Learning cites Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Hard Negative Mixing for Contrastive Learning cites Debiased Contrastive Learning
- SimCSE: Simple Contrastive Learning of Sentence Embeddings cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Debiased Contrastive Learning cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
Cited by top-tier papers1
Builds on16
- Unsupervised Learning of Visual Features by Contrasting Cluster AssignmentsMathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal et al.NeurIPS 2020 · 5,249 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the HypersphereTongzhou Wang, Phillip IsolaICML 2020 · 2,360 citations
- Hard Negative Mixing for Contrastive LearningYannis Kalantidis, Mert Bülent Sariyildiz, Noé Pion, Philippe Weinzaepfel et al.NeurIPS 2020 · 805 citations
- Debiased Contrastive LearningChing-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba et al.NeurIPS 2020 · 761 citations
Related papers
- Unsupervised Sentence Representation via Contrastive Learning with Mixing NegativesYanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu et al.AAAI 2022 · 71 citations
- Debiased Contrastive Learning of Unsupervised Sentence RepresentationsKun Zhou, Beichen Zhang, Wayne Xin Zhao, Ji-Rong WenACL 2022 · 128 citations
- Contrastive Learning with Hard Negative SamplesJoshua David Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie JegelkaICLR 2021 · 999 citations
- InfoCSE: Information-aggregated Contrastive Learning of Sentence EmbeddingsXing Wu, Chaochen Gao, Zijia Lin, Jizhong Han et al.EMNLP Findings 2022 · 24 citations
- ProGCL: Rethinking Hard Negative Mining in Graph Contrastive LearningJun Xia, Lirong Wu, Ge Wang, Jintao Chen et al.ICML 2022 · 174 citations
