Dynamically Masked Discriminator for GANs

Training Generative Adversarial Networks (GANs) remains a challenging problem. The discriminator trains the generator by learning the distribution of real/generated data. However, the distribution of generated data changes throughout the training process, which is difficult for the discriminator to learn. In this paper, we propose a novel method for GANs from the viewpoint of online continual learning. We observe that the discriminator model, trained on historically generated data, often slows down its adaptation to the changes in the new arrival generated data, which accordingly decreases the quality of generated results. By treating the generated data in training as a stream, we propose to detect whether the discriminator slows down the learning of new knowledge in generated data. Therefore, we can explicitly enforce the discriminator to learn new knowledge fast. Particularly, we propose a new discriminator, which automatically detects its retardation and then dynamically masks its features, such that the discriminator can adaptively learn the temporally-vary distribution of generated data. Experimental results show our method outperforms the state-of-the-art approaches. Introduction Generative Adversarial Networks (GANs) [19, 84, 20, 5, 72, 70, 71] have shown the remarkable performance in various applications [89, 85, 81, 3, 44] , which have attracted intensive interests. The main components of GANs are the generator and the discriminator, where the generator is trained to generate realistic samples, and the discriminator learns to distinguish real and generated samples. However, it is well-known that GANs are difficult to train [33, 30, 21] . Many efforts have been devoted to alleviating training difficulties for GANs. Some studies balance the generator and discriminator from the side of network architecture, such as DCGAN [61], PG-GAN [32], 36, 34], and BigGAN [6]. Besides, some studies [2, 94, 53, 86] address this challenge from the learning objective, e.g., WGAN [2], EBGAN [94], LSGAN[53] and realness-GAN [86] . These methods can further stabilize training, especially coupled with the Lipschitz regularization [56, 21] and conditional information [55] . Recent works [87, 33, 30, 95, 93, 50, 48, 82] propose to improve GANs on the discriminator side. For example, different data augmentation strategies [33, 30, 95] are proposed to strengthen the discriminator for a limited data regime. Regularizations are applied to stabilize the training of discriminator [93, 50, 82] and combat the mode † Equal Contribution. Haozhe worked on this project before joining KAUST.