Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes
Abstract
We derive non-asymptotic spectral bands that bound the squared InfoNCE gradient norm via alignment, temperature, and batch spectrum, recovering the law and closely tracking batch-mean gradients on synthetic data and ImageNet. Using effective rank as an anisotropy proxy, we design spectrum-aware batch selection, including a fast greedy builder. On ImageNet-100, Greedy-64 cuts time-to-67.5% top-1 by 15% vs. random (24% vs. Pool--P3) at equal accuracy; CIFAR-10 shows similar gains. In-batch whitening promotes isotropy and reduces 50-step gradient variance by , matching our theoretical upper bound.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere cites A Simple Framework for Contrastive Learning of Visual Representations
- Contrastive Learning with Hard Negative Samples cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Contrastive Learning with Hard Negative Samples cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Supervised Contrastive Learning cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
- Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
Builds on14
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
- Understanding Contrastive Representation Learning through Alignment and Uniformity on the HypersphereTongzhou Wang, Phillip IsolaICML 2020 · 2,360 citations
- Contrastive Learning with Hard Negative SamplesJoshua David Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie JegelkaICLR 2021 · 999 citations
Related papers
- Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase RetrievalGuillaume Braun, Han Bao, Wei Huang, Masaaki ImaizumiICML 2026
- Improving Generalization of Batch Whitening by Convolutional Unit OptimizationYooshin Cho, Hanbyel Cho, Youngsoo Kim, Junmo KimICCV 2021 · 3 citations
- Beyond Soft Label: Dataset Distillation via Orthogonal Gradient MatchingDeyu Bo, Xinchao WangCVPR 2026
- Spectral Bias in Practice: The Role of Function Frequency in GeneralizationSara Fridovich-Keil, Raphael Gontijo Lopes, Rebecca RoelofsNeurIPS 2022 · 61 citations
- Group Whitening: Balancing Learning Efficiency and Representational CapacityLei Huang, Yi Zhou, Li Liu, Fan Zhu et al.CVPR 2021
