High-Order Residual Network for Light Field Super-Resolution

Plenoptic cameras usually sacrifice the spatial resolution of their SAIs to acquire geometry information from different viewpoints. Several methods have been proposed to mitigate such spatio-angular trade-off, but seldom make use of the structural properties of the light field (LF) data efficiently. In this paper, we propose a novel high-order residual network to learn the geometric features hierarchically from the LF for reconstruction. An important component in the proposed network is the high-order residual block (HRB), which learns the local geometric features by considering the information from all input views. After fully obtaining the local features learned from each HRB, our model extracts the representative geometric features for spatio-angular upsampling through the global residual learning. Additionally, a refinement network is followed to further enhance the spatial details by minimizing a perceptual loss. Compared with previous work, our model is tailored to the rich structure inherent in the LF, and therefore can reduce the artifacts near non-Lambertian and occlusion regions. Experimental results show that our approach enables high-quality reconstruction even in challenging regions and outperforms state-of-the-art single image or LF reconstruction methods with both quantitative measurements and visual evaluation. Compared to a 2D imaging system, the plenoptic camera not only captures the accumulated intensity of a light ray at each point in space, but also provides the directional radiance information. Together they form the light field (LF), which has shown advantages over 2D imagery in problems such as disparity estimation (Jeon et al. 2015; Sun et al. 2016) or 3D reconstruction (Heber, Yu, and Pock 2017) of a scene, generation of images for a novel viewpoint (Kalantari, Wang, and Ramamoorthi 2016; Meng et al. 2019b), and refocusing (Mitra and Veeraraghavan 2012). Nevertheless, in practice, it can be difficult to achieve a dense sampling of the entire LF due to the limited resolution of the camera sensor. Acquisition of a densely sampled subaperture image (SAI) usually sacrifices the view point information, or vice versa (Wanner and Goldluecke 2014). As a result, the LF views exhibit a lower spatial resolution than * This work was done during an internship at Huawei.