Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks

Pre-training on large-scale datasets and then fine-tuning on downstream tasks have become a standard practice in deep learning. However, pre-training datasets, while inaccessible or too expensive to handle, often contain label noise that may adversely affect the generalization of the model and pose unexpected risks. This paper aims to understand the nature of noise in pre-training datasets and then mitigate its impact on downstream tasks. Specifically, through extensive experiments of supervised pre-training models on synthetic noisy ImageNet-1K and YFCC15M datasets, we demonstrate that while slight noise in pre-training can benefit in-domain (ID) performance, where the training and testing data share the same distribution, it always deteriorates out-of-domain (OOD) performance, where training and testing distributions are different. We empirically ascertain that the reason behind is noise in pre-training shapes the feature space differently. We then propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization on both ID and OOD tasks, considering that one may not be able to access or fully fine-tune the pre-trained models. We conduct extensive experiments on popular vision and language models including APIs that are supervised and self-supervised pre-trained on real data for evaluation. Our results show the importance of this novel and fundamental research direction, which we term Noisy Model Learning.