Training Pansharpening Networks at Full Resolution Using Degenerate Invariance

Pansharpening is an important technique for remote sensing imaging systems to obtain high-resolution multispectral images. Existing deep learning-based methods mostly rely on using pseudo-groundtruth multi-spectral images for supervised learning. The whole training process only remains at the scale of reduced resolution, which means that the impact of the degradation process is ignored and high-quality images cannot be guaranteed at full resolution. To address the challenge, we propose a new unsupervised framework that does not rely on pseudo-groundtruth but uses the invariance of the degradation process to build a consistent loss function on the original scale for network training. Specifically, we first introduce the operator learning method to build an exact mapping function from multi-spectral to panchromatic images and decouple both spectral and texture features. Then, through joint training, operators and convolutional networks can learn the spatial degradation process and spectral degradation process at full resolution, respectively. By introducing them to build consistency constraints, we can train the pansharpening network at the original full resolution. Our approach can be applied to existing pansharpening methods, improving their usability on original data, which matches practical application requirements. The experimental results on different kinds of satellite datasets demonstrate that the proposed network outperforms state-of-the-art methods both visually and quantitatively. Our code is available at https://github.com/quycruin/Qvac.