Lune

NeurIPS2025Top-tier venue

Diversity Is All You Need for Contrastive Learning: Spectral Bounds on Gradient Magnitudes

Peter Ochieng

Abstract

We derive non-asymptotic spectral bands that bound the squared InfoNCE gradient norm via alignment, temperature, and batch spectrum, recovering the 1/τ21/\tau^{2} law and closely tracking batch-mean gradients on synthetic data and ImageNet. Using effective rank ReffR_{\mathrm{eff}} as an anisotropy proxy, we design spectrum-aware batch selection, including a fast greedy builder. On ImageNet-100, Greedy-64 cuts time-to-67.5% top-1 by 15% vs. random (24% vs. Pool--P3) at equal accuracy; CIFAR-10 shows similar gains. In-batch whitening promotes isotropy and reduces 50-step gradient variance by 1.37×1.37\times, matching our theoretical upper bound.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere cites A Simple Framework for Contrastive Learning of Visual Representations
  • Contrastive Learning with Hard Negative Samples cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
  • Contrastive Learning with Hard Negative Samples cites A Simple Framework for Contrastive Learning of Visual Representations
  • Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
  • Supervised Contrastive Learning cites Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
  • Supervised Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  2. Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
  3. Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
  4. Understanding Contrastive Representation Learning through Alignment and Uniformity on the HypersphereTongzhou Wang, Phillip IsolaICML 2020 · 2,360 citations
  5. Contrastive Learning with Hard Negative SamplesJoshua David Robinson, Ching-Yao Chuang, Suvrit Sra, Stefanie JegelkaICLR 2021 · 999 citations

Related papers

  1. Spectral Gradient Descent Mitigates Anisotropy-Driven Misalignment: A Case Study in Phase RetrievalGuillaume Braun, Han Bao, Wei Huang, Masaaki ImaizumiICML 2026
  2. Improving Generalization of Batch Whitening by Convolutional Unit OptimizationYooshin Cho, Hanbyel Cho, Youngsoo Kim, Junmo KimICCV 2021 · 3 citations
  3. Beyond Soft Label: Dataset Distillation via Orthogonal Gradient MatchingDeyu Bo, Xinchao WangCVPR 2026
  4. Spectral Bias in Practice: The Role of Function Frequency in GeneralizationSara Fridovich-Keil, Raphael Gontijo Lopes, Rebecca RoelofsNeurIPS 2022 · 61 citations
  5. Group Whitening: Balancing Learning Efficiency and Representational CapacityLei Huang, Yi Zhou, Li Liu, Fan Zhu et al.CVPR 2021
Dusk over the sea between two cliffs drawn in fine vertical lines