How Well Do Self-Supervised Models Transfer?
Abstract
InsDis MoCo-v1 PCL-v1 PIRL PCL-v2 SimCLR-v1 MoCo-v2 SimCLR-v2 SeLa-v2 InfoMin BYOL DeepCluster-v2 SwAV Supervised Figure 1 . Transfer performance is highly correlated with ImageNet performance for many-shot recognition but increasingly less correlated for few-shot recognition, object detection and dense prediction. On the x-axes we plot ImageNet top-1 accuracy and on the y-axes the average transfer log-odds. The gradients of the regression lines describe the correlation, with confidence intervals in shaded areas. For perfect correlation, the ideal line is a positive slope diagonal. Correlation coefficients (Pearson's r) are shown in the top left of each plot.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
81 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Improving CLIP Training with Language Rewrites cites A Simple Framework for Contrastive Learning of Visual Representations
- Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference cites Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference cites A Simple Framework for Contrastive Learning of Visual Representations
- Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference cites Big Self-Supervised Models are Strong Semi-Supervised Learners
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments cites A Simple Framework for Contrastive Learning of Visual Representations
- StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners cites A Simple Framework for Contrastive Learning of Visual Representations
- StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners cites What Makes for Good Views for Contrastive Learning?
- CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion cites Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion cites A Simple Framework for Contrastive Learning of Visual Representations
- CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion cites Big Self-Supervised Models are Strong Semi-Supervised Learners
- CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Big Self-Supervised Models are Strong Semi-Supervised Learners cites A Simple Framework for Contrastive Learning of Visual Representations
- Big Self-Supervised Models are Strong Semi-Supervised Learners cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites Big Self-Supervised Models are Strong Semi-Supervised Learners
- Big Self-Supervised Models are Strong Semi-Supervised Learners cites What Makes for Good Views for Contrastive Learning?
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites What Makes for Good Views for Contrastive Learning?
- Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods cites A Simple Framework for Contrastive Learning of Visual Representations
- What Makes for Good Views for Contrastive Learning? cites A Simple Framework for Contrastive Learning of Visual Representations
Cited by top-tier papers81
- Improving CLIP Training with Language RewritesLijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi et al.NeurIPS 2023 · 308 citations
- StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation LearnersYonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang et al.NeurIPS 2023 · 251 citations
- CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View CompletionPhilippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier et al.NeurIPS 2022 · 189 citations
- Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding MethodsRandall Balestriero, Yann LeCunNeurIPS 2022 · 189 citations
- Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a DifferenceShell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim et al.CVPR 2022 · 161 citations
Builds on13
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Unsupervised Learning of Visual Features by Contrasting Cluster AssignmentsMathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal et al.NeurIPS 2020 · 5,249 citations
- Big Self-Supervised Models are Strong Semi-Supervised LearnersTing Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi et al.NeurIPS 2020 · 2,611 citations
- What Makes for Good Views for Contrastive Learning?Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan et al.NeurIPS 2020 · 1,631 citations
Related papers
- Does Robustness on ImageNet Transfer to Downstream Tasks?Yutaro Yamada, Mayu OtaniCVPR 2022 · 23 citations
- Do Adversarially Robust ImageNet Models Transfer Better?Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor et al.NeurIPS 2020 · 506 citations
- Exploring Effective Knowledge Transfer for Few-shot Object DetectionZhiyuan Zhao, Qingjie Liu, Yunhong WangACM MM 2022 · 16 citations
- CrossTransformers: spatially-aware few-shot transferCarl Doersch, Ankush Gupta, Andrew ZissermanNeurIPS 2020 · 420 citations
- Accuracy on the Line: on the Strong Correlation Between Out-of-Distribution and In-Distribution GeneralizationJohn Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa et al.ICML 2021 · 323 citations
