Flow-Based Kernel Prior With Application to Blind Super-Resolution

Kernel estimation is generally one of the key problems for blind image super-resolution (SR). Recently, Double-DIP proposes to model the kernel via a network architecture prior, while KernelGAN employs the deep linear network and several regularization losses to constrain the kernel space. However, they fail to fully exploit the general SR kernel assumption that anisotropic Gaussian kernels are sufficient for image SR. To address this issue, this paper proposes a normalizing flow-based kernel prior (FKP) for kernel modeling. By learning an invertible mapping between the anisotropic Gaussian kernel distribution and a tractable latent distribution, FKP can be easily used to replace the kernel modeling modules of Double-DIP and Ker-nelGAN. Specifically, FKP optimizes the kernel in the latent space rather than the network parameter space, which allows it to generate reasonable kernel initialization, traverse the learned kernel manifold and improve the optimization stability. Extensive experiments on synthetic and realworld images demonstrate that the proposed FKP can significantly improve the kernel estimation accuracy with less parameters, runtime and memory usage, leading to stateof-the-art blind SR results. * Corresponding author. kernels is becoming an active research topic. Compared to non-blind SR, blind SR generally needs to additionally estimate the blur kernel and thus is more illposed. A popular line of work tries to decompose blind SR into two sub-problems, i.e., kernel estimation and nonblind SR. As a preliminary step of non-blind SR, kernel estimation plays a crucial role. If the estimated kernel deviates from the ground-truth, the HR image reconstructed by the non-blind SR methods would deteriorate seriously [11, 17, 50] . In view of this, this paper focuses on the SR kernel estimation problem. Recently, some kernel estimation methods, such as Double-DIP [15, 39] and KernelGAN [3], have shown promising results. Specifically, with two deep image priors (DIPs) [44] , Double-DIP can be used to jointly optimize the HR image and blur kernel in the parameter space of untrained encoder-decoder networks by minimizing the LR image reconstruction error. Although DIP has shown to be effective for modeling natural images, whether it is effective to model blur kernel or not remains unclear. The main reason is that blur kernel usually has a small spatial size and has its own characteristics that differ from natural images. In [39], a fully-connected network (FCN) is used to model the kernel prior, which, however, lacks interpretability. With a different framework to Double-DIP, KernelGAN designs an internal generative adversarial network (GAN) for the LR image on the basis of image patch recurrence property [16, 35, 57] . It defines the kernel implicitly by a deep linear network, which is optimized by the GAN loss and five extra regularization losses such as sparsity loss. Obviously, these two methods do not make full use of the anisotropic Gaussian kernel prior which has been demonstrated to be effective enough for real image SR [11, 40, 50, 54, 55] . In this paper, we propose a flow-based kernel prior (FKP) for kernel distribution modeling and incorporate it into existing blind SR models. Based on normalizing flow, FKP consists of several batch normalization layers, permutation layers and affine coupling layers, which allow the model