DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data

Generative adversarial nets (GANs) have been remarkably successful at learning to sample from distributions specified by a given dataset, particularly if the given dataset is reasonably large compared to its dimensionality. However, given limited data, classical GANs have struggled, and strategies like output-regularization, data-augmentation, use of pre-trained models and pruning have been shown to lead to improvements. Notably, the applicability of these strategies is 1) often constrained to particular settings, e.g., availability of a pretrained GAN; or 2) increases training time, e.g., when using pruning. In contrast, we propose a Discriminator gradIent Gap regularized GAN (DigGAN) formulation which can be added to any existing GAN. DigGAN augments existing GANs by encouraging to narrow the gap between the norm of the gradient of a discriminator's prediction w.r.t. real images and w.r.t. the generated samples. We observe this formulation to avoid bad attractors within the GAN loss landscape, and we find DigGAN to significantly improve the results of GAN training when limited data is available. Code is available at https://github.com/AilsaF/DigGAN . Introduction Generative Adversarial Nets (GANs) [13] have been remarkably successful at learning to sample from distributions specified by a given dataset. In practice, this success has garnered a lot of interest in GANs for a wide range of applications, from data augmentation [23, 61] and domain adaptation [8, 48] to image-to-image translation [63, 18, 24] and photo editing [4, 64] . This success of GANs strongly relies on the availability of a large dataset. Unsurprisingly, in real-life circumstances, particularly when the dimensionality of the samples in the dataset is high, the available samples to train a GAN can be insufficient. Insufficient data may significantly reduce the performance of standard GANs. For instance, when we train a GAN on CIFAR-100 using just 10% of the data, BigGAN performance deteriorates from 13.54 FID score to 73.01 FID score, and the GAN generates images of a single pattern (Fig. 1 ). To address this deteriorating performance of GANs trained with limited data, various strategies have been proposed recently, including the use of a pretrained model [60, 42, 28] , pruning [7] , and data augmentation [23, 61, 59, 62, 19] . However, despite improving results, each of these strategies also imposes restrictions. The use of pretrained models works best if data domains remain similar. Pruning requires many rounds of training to increase the sparsity of the neural architecture, which raises the training cost. Data augmentation can enhance the results, but the benefit is limited with insufficient data (Tab. 3). Regularization is a cheap and potentially effective approach, and recent work by Tseng et al. [51] adopted this approach, controlling the distance between the discriminator's prediction on the real image and the generated image. However, with limited data, this regularization doesn't show significant improvements (Tab. 3). In this paper, we study a new regularization to enhance the training of GANs with limited data. Instead of constraining the discriminator's output as done in prior work [51] , we propose the Discriminator 36th Conference on Neural Information Processing Systems (NeurIPS 2022).