Lune

NeurIPS2020Top-tier venue

Hard Negative Mixing for Contrastive Learning

Yannis Kalantidis, Mert Bülent Sariyildiz, Noé Pion, Philippe Weinzaepfel, Diane Larlus

Abstract

Contrastive learning has become a key component of self-supervised learning approaches for computer vision. By learning to embed two augmented versions of the same image close to each other and to push the embeddings of different images apart, one can train highly transferable visual representations. As revealed by recent studies, heavy data augmentation and large sets of negatives are both crucial in learning such representations. At the same time, data mixing strategies, either at the image or the feature level, improve both supervised and semi-supervised learning by synthesizing novel examples, forcing networks to learn more robust features. In this paper, we argue that an important aspect of contrastive learning, i.e. the effect of hard negatives, has so far been neglected. To get more meaningful negative samples, current top contrastive self-supervised learning approaches either substantially increase the batch sizes, or keep very large memory banks; increasing memory requirements, however, leads to diminishing returns in terms of performance. We therefore start by delving deeper into a top-performing framework and show evidence that harder negatives are needed to facilitate better and faster learning. Based on these observations, and motivated by the success of data mixing, we propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead. We exhaustively ablate our approach on linear classification, object detection, and instance segmentation and show that employing our hard negative mixing procedure improves the quality of visual representations learned by a state-of-the-art self-supervised learning method.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

179 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Support-set bottlenecks for video-text representation learning cites Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
  • Support-set bottlenecks for video-text representation learning cites A Simple Framework for Contrastive Learning of Visual Representations
  • Unsupervised Learning of Visual Features by Contrasting Cluster Assignments cites A Simple Framework for Contrastive Learning of Visual Representations
  • Unsupervised Learning of Visual Features by Contrasting Cluster Assignments cites Supervised Contrastive Learning
  • CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding cites A Simple Framework for Contrastive Learning of Visual Representations
  • CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
  • Contrastive Learning with Hard Negative Samples cites A Simple Framework for Contrastive Learning of Visual Representations
  • Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling cites Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
  • Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
  • Supervised Contrastive Learning cites CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features
  • Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
  • Exploring Cross-Image Pixel Contrast for Semantic Segmentation cites Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
  • Exploring Cross-Image Pixel Contrast for Semantic Segmentation cites A Simple Framework for Contrastive Learning of Visual Representations
  • Exploring Cross-Image Pixel Contrast for Semantic Segmentation cites Contrastive Learning with Hard Negative Samples
  • Exploring Cross-Image Pixel Contrast for Semantic Segmentation cites Supervised Contrastive Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers179

  1. Contrastive Learning with Hard Negative SamplesJoshua David Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie JegelkaICLR 2021 · 999 citations
  2. Exploring Cross-Image Pixel Contrast for Semantic SegmentationWenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai et al.ICCV 2021 · 568 citations
  3. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware SamplingSebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin et al.SIGIR 2021 · 297 citations
  4. CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud UnderstandingMohamed Afham, Isuru Dissanayake, Dinithi Dissanayake, Amaya Dharmasiri et al.CVPR 2022 · 286 citations
  5. Support-set bottlenecks for video-text representation learningMandela Patrick, Po-Yao Huang, Yuki Markus Asano, Florian Metze et al.ICLR 2021 · 269 citations

Builds on38

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
  3. Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
  4. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable FeaturesSangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh et al.ICCV 2019 · 5,843 citations
  5. Unsupervised Learning of Visual Features by Contrasting Cluster AssignmentsMathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal et al.NeurIPS 2020 · 5,249 citations

Related papers

  1. Unsupervised Sentence Representation via Contrastive Learning with Mixing NegativesYanzhao Zhang, Richong Zhang, Samuel Mensah, Xudong Liu et al.AAAI 2022 · 71 citations
  2. Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory BankIñigo Alonso, Alberto Sabater, David Ferstl, Luis Montesano et al.ICCV 2021 · 261 citations
  3. Bootstrapping Semantic Segmentation with Regional ContrastShikun Liu, Shuaifeng Zhi, Edward Johns, Andrew J. DavisonICLR 2022 · 151 citations
  4. SemPPL: Predicting Pseudo-Labels for Better Contrastive RepresentationsMatko Bosnjak, Pierre Harvey Richemond, Nenad Tomasev, Florian Strub et al.ICLR 2023 · 4 citations
  5. Incremental False Negative Detection for Contrastive LearningTsai-Shien Chen, Wei-Chih Hung, Hung-Yu Tseng, Shao-Yi Chien et al.ICLR 2022 · 82 citations
Dusk over the sea between two cliffs drawn in fine vertical lines