Understanding Over-parameterization in Generative Adversarial Networks
Yogesh Balaji, Mohammadmahdi Sajedi, Neha Mukund Kalibhat, Mucong Ding, Dominik Stöger, Mahdi Soltanolkotabi, Soheil Feizi
Abstract
A broad class of unsupervised deep learning methods such as Generative Adversarial Networks (GANs) involve training of overparameterized models where the number of parameters of the model exceeds a certain threshold. Indeed, most successful GANs used in practice are trained using overparameterized generator and discriminator networks, both in terms of depth and width. A large body of work in supervised learning have shown the importance of model overparameterization in the convergence of the gradient descent (GD) to globally optimal solutions. In contrast, the unsupervised setting and GANs in particular involve non-convex concave mini-max optimization problems that are often trained using Gradient Descent/Ascent (GDA). The role and benefits of model overparameterization in the convergence of GDA to a global saddle point in non-convex concave problems is far less understood. In this work, we present a comprehensive analysis of the importance of model overparameterization in GANs both theoretically and empirically. We theoretically show that in an overparameterized GAN model with a 1-layer neural network generator and a linear discriminator, GDA converges to a global saddle point of the underlying non-convex concave min-max problem. To the best of our knowledge, this is the first result for global convergence of GDA in such settings. Our theory is based on a more general result that holds for a broader class of nonlinear generators and discriminators that obey certain assumptions (including deeper generators and random feature discriminators). Our theory utilizes and builds upon a novel connection with the convergence analysis of linear timevarying dynamical systems which may have broader implications for understanding the convergence behavior of GDA for non-convex concave problems involving overparameterized models. We also empirically study the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and Celeb-A datasets. Our experiments show that overparameterization improves the quality of generated samples across various model architectures and datasets. Remarkably, we observe that overparameterization leads to faster and more stable convergence behavior of GDA across the board.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Instability and Local Minima in GAN Training with Kernel Discriminators cites A Neural Tangent Kernel Perspective of GANs
- A Neural Tangent Kernel Perspective of GANs cites Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form Solutions
Cited by top-tier papers9
- Practical GAN-based synthetic IP header trace generation using NetShareYucheng Yin, Zinan Lin, Minhao Jin, Giulia Fanti et al.SIGCOMM 2022 · 106 citations
- InfinityGAN: Towards Infinite-Pixel Image SynthesisChieh Hubert Lin, Hsin-Ying Lee, Yen-Chi Cheng, Sergey Tulyakov et al.ICLR 2022 · 84 citations
- A Neural Tangent Kernel Perspective of GANsJean-Yves Franceschi, Emmanuel de Bézenac, Ibrahim Ayed, Mickaël Chen et al.ICML 2022 · 29 citations
- Hidden Convexity of Wasserstein GANs: Interpretable Generative Models with Closed-Form SolutionsArda Sahiner, Tolga Ergen, Batu Ozturkler, Burak Bartan et al.ICLR 2022 · 23 citations
- Instability and Local Minima in GAN Training with Kernel DiscriminatorsEvan Becker, Parthe Pandit, Sundeep Rangan, Alyson K. FletcherNeurIPS 2022 · 18 citations
Builds on3
- A Closer Look at the Optimization Landscapes of Generative Adversarial NetworksHugo Berard, Gauthier Gidel, Amjad Almahairi, Pascal Vincent et al.ICLR 2020 · 66 citations
- SGD Learns One-Layer Networks in WGANsQi Lei, Jason D. Lee, Alex Dimakis, Constantinos DaskalakisICML 2020 · 36 citations
- The complexity of constrained min-max optimizationConstantinos Daskalakis, Stratis Skoulakis, Manolis ZampetakisSTOC 2021 · 18 citations
Related papers
- Forward Super-Resolution: How Can GANs Learn Hierarchical Generative Models for Real-World DistributionsZeyuan Allen-Zhu, Yuanzhi LiICLR 2023 · 4 citations
- On Convergence of Gradient Descent Ascent: A Tight Local AnalysisHaochuan Li, Farzan Farnia, Subhro Das, Ali JadbabaieICML 2022 · 12 citations
- On Gradient Descent Ascent for Nonconvex-Concave Minimax ProblemsTianyi Lin, Chi Jin, Michael I. JordanICML 2020 · 587 citations
- A Decentralized Parallel Algorithm for Training Generative Adversarial NetsMingrui Liu, Wei Zhang, Youssef Mroueh, Xiaodong Cui et al.NeurIPS 2020 · 6 citations
- Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial NetsMingrui Liu, Youssef Mroueh, Jerret Ross, Wei Zhang et al.ICLR 2020 · 67 citations
