Your GAN is Secretly an Energy-based Model and You Should Use Discriminator Driven Latent Sampling

The sum of the implicit generator log-density log p g of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal. This makes it possible to improve on the typical generator (with implicit density p g ). We show that samples can be generated from this modified density by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. We call this process of running Markov Chain Monte Carlo in the latent space, and then applying the generator function, Discriminator Driven Latent Sampling (DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space, and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN [1] from 8.22 to 9.09 which is comparable to the class-conditional BigGAN [2] model. This achieves a new state-of-the-art in the unconditional image synthesis setting without introducing extra parameters or additional training. * Equal contribution. Ordering determined by coin flip. 34th Conference on Neural Information Processing Systems (NeurIPS 2020),