The Close Relationship Between Contrastive Learning and Meta-Learning

Contrastive learning has recently taken off as a paradigm for learning from unlabeled data. In this paper, we discuss the close relationship between contrastive learning and meta-learning under a certain task distribution. We complement this observation by showing that established meta-learning methods, such as Prototypical Networks, achieve comparable performance to SimCLR when paired with this task distribution. This relationship can be leveraged by taking established techniques from meta-learning, such as task-based data augmentation, and showing that they benefit contrastive learning as well. These tricks also benefit state-of-the-art self-supervised learners without using negative pairs such as BYOL, which achieves 94.6% accuracy on CIFAR-10 using a self-supervised ResNet-18 feature extractor trained with our meta-learning tricks. We conclude that existing advances designed for contrastive learning or metalearning can be exploited to benefit the other, and it is better for contrastive learning researchers to take lessons from the meta-learning literature (and viceversa) than to reinvent the wheel. Our Pytorch implementation can be found on: https://github.com/RenkunNi/MetaContrastive