Large-Margin Contrastive Learning with Distance Polarization Regularizer

Contrastive learning (CL) pretrains models in a pairwise manner, where given a data point, other data points are all regarded as dissimilar, including some that are semantically similar. The issue has been addressed by properly weighting similar and dissimilar pairs as in positive-unlabeled learning, so that the objective of CL is unbiased and CL is consistent. However, in this paper, we argue that this great solution is still not enough: its weighted objective hides the issue where the semantically similar pairs are still pushed away; as CL is pretraining, this phenomenon is not our desideratum and might affect downstream tasks. To this end, we propose large-margin contrastive learning (LMCL) with distance polarization regularizer, motivated by the distribution characteristic of pairwise distances in metric learning. In LMCL, we can distinguish between intra-cluster and inter-cluster pairs, and then only push away inter-cluster pairs, which solves the above issue explicitly. Theoretically, we prove a tighter error bound for LMCL; empirically, the superiority of LMCL is demonstrated across multiple domains, i.e., image classification, sentence representation, and reinforcement learning.