Uncertainty-Aware GAN for Single Image Super Resolution

Generative adversarial network (GAN) has become a popular tool in the perception-oriented single image super-resolution (SISR) for its excellent capability to hallucinate details. However, the performance of most GAN-based SISR methods is impeded due to the limited discriminative ability of their discriminators. In specific, these discriminators only focus on the global image reconstruction quality and ignore the more fine-grained reconstruction quality for constraining the generator, as they predict the overall realness of an image instead of the pixel-level realness. Here, we first introduce the uncertainty into the GAN and propose an Uncertainty-aware GAN (UGAN) to regularize SISR solutions, where the challenging pixels with large reconstruction uncertainty and importance (e.g., texture and edge) are prioritized for optimization. The uncertainty-aware adversarial training strategy enables the discriminator to capture the pixel-level SR uncertainty, which constrains the generator to focus on image regions with high reconstruction difficulty, meanwhile, it improves the interpretability of the SR. To balance the weights of multiple training losses, we introduce an uncertainty-aware loss weighting strategy to learn the optimal loss weights adaptively. Extensive experiments demonstrate the effectiveness of our approach in extracting the SR uncertainty and the superiority of the UGAN over the state-of-the-art in terms of reconstruction accuracy and perceptual quality.