Waxing-and-Waning: a Generic Similarity-based Framework for Efficient Self-Supervised Learning

Deep Neural Networks (DNNs), essential for diverse applications such as visual recognition and eldercare, often require a large amount of labeled data for training, making widespread deployment of DNNs a challenging task. Self-supervised learning (SSL) emerges as a promising approach, which leverages inherent patterns within data through diverse augmentations to train models without explicit labels. However, while SSL has shown notable advancements in accuracy, its high computation costs remain a daunting impediment, particularly for resourceconstrained platforms. To address this problem, we introduce SIMWNW, a similarity-based efficient self-supervised learning framework. By strategically removing less important regions in augmented images and feature maps, SIMWNW not only reduces computation costs but also eliminates irrelevant features that might slow down the learning process, thereby accelerating model convergence. The experimental results show that SIMWNW effectively reduces the amount of computation costs in self-supervised model training without compromising accuracy. Specifically, SIMWNW yields up to 54% and 51% computation savings in training from scratch and transfer learning tasks, respectively.

Published as a conference paper at ICLR 2024 efficiency. So, it is natural to raise a question: Is there a more general and effective method that can significantly improve the training efficiency of SSL?

Considering the training paradigm of the SSL that leverages different data augmentations on two branches with a siamese encoder model used, it results in a unique property of SSL compared to the conventional supervised learning methods (Tian et al., 2020; Chen et al., 2020a). That is, the augmented input images and feature maps on the two branches inherently have a certain degree of similarity. This is a natural opportunity that could be potentially used for computation saving or simplifying. However, it is insufficient to lead us directly to a simple solution. It is not clear whether similar and dissimilar regions of augmented input images and feature maps in the two branches are equivalently crucial for SSL and whether the similarity remains invariant in low-level features and high-level features. And how can we effectively utilize the similarity to improve training efficiency? Motivated by these questions, we make a comprehensive exploration of the impact of similar regions on SSL accuracy. And we explore two types of methods (i.e., reuse and remove) to exploit the similarities for computation-saving. We find that eliminating the computation on similar regions of augmented input images and each layer's activations can significantly reduce the computation and speed up SSL. To mitigate the region shrinking problem caused by convolution layers, we propose a strategy to effectively and efficiently identify and expand high-similarity regions to ensure a decent overall computation saving. This strategy can be considered a waxing-and-waning process. Putting it all together, we propose our SIMWNW, a generic and efficient SSL framework that can significantly reduce training costs and improve the convergence speed of SSL.

Our SIMWNW framework is generic and can be easily applied to different SSL training methods for training cost-saving. We evaluate our framework in both training from scratch and transfer learning tasks and validate the effectiveness and generalizability of SIMWNW. Specifically, in training from scratch tasks, compared to representative SSL works, SIMWNW provides significant computation savings, peaking at 54% and averaging at 40%, and without sacrificing accuracy. In transfer learning tasks, SIMWNW shows a notable reduction in computation costs, peaking at 51% and averaging at 48%, without accuracy loss. We also compare SIMWNW to efficient SSL approaches. In both training from scratch and transfer learning tasks, SIMWNW consistently outperforms SOTA works, achieving an average computational cost reduction of 18% and 14%, respectively.