AUC-CL: A Batchsize-Robust Framework for Self-Supervised Contrastive Representation Learning

Self-supervised learning through contrastive representations is an emergent and promising avenue, aiming at alleviating the availability of labeled data. Recent research in the field also demonstrates its viability for several downstream tasks, henceforth leading to works that implement the contrastive principle through innovative loss functions and methods. However, despite achieving impressive progress, most methods depend on prohibitively large batch sizes and compute requirements for good performance. In this work, we propose the AUC-Contrastive Learning, a new approach to contrastive learning that demonstrates robust and competitive performance in compute-limited regimes. We propose to incorporate the contrastive objective within the AUC-maximization framework, by noting that the AUC metric is maximized upon enhancing the probability of the network's binary prediction difference between positive and negative samples which inspires adequate embedding space arrangements in representation learning. Unlike standard contrastive methods, when performing stochastic optimization, our method maintains unbiased stochastic gradients and thus is more robust to batchsizes as opposed to standard stochastic optimization problems. Remarkably, our method with a batch size of 256, outperforms several state-of-the-art methods that may need much larger batch sizes (e.g., 4096), on ImageNet and other standard datasets. Experiments on transfer learning and few-shot learning tasks also demonstrate the downstream viability of our method. Code is available at AUC-CL. Recent innovations falling under image sample contrastive methods that we consider relevant to this work include the SimCLR (Chen et al., 2020 ), MoCo v3 (Chen et al., 2021), CLIP (Radford et al., 2021), DCL (Yeh et al., 2022) and others, which utilize variants of the cross-entropy criterion. A prominent characteristic of these works is the requirement for large batch sizes in order to enhance performance, which often mandates a steep compute requirement and training time. CONTRASTIVE LEARNING AND ITS LIMITATIONS Contrastive Learning can date back to work in the early 90's (Becker & Hinton, 1992; Bromley et al., 1993) , based on the motive of utilization of internally derived teaching signals in a neural