Towards Event-guided Panoramic HDR Video Reconstruction for Indoor Immersive VR: A Novel Dataset and Approach

High Dynamic Range (HDR) panoramic video is crucial to enhance immersive experience in Virtual Reality (VR). However, a hurdle is that panoramic cameras often struggle with limited dynamic range and motion blur. Inspired by the event-driven sensing of the human eye, this paper explores the potential of the event cameras to enhance panoramic HDR video reconstruction. As a pioneering research endeavor, we first starts by designing a novel hybrid imaging platform equipped with preprocessing pipelines for event-panorama synchronization, alignment, and HDR ground truth generation. Based on the platform, we then introduce Ev-Pano, the first event-panorama HDR video covering diverse indoor scenes for panoramic HDR video reconstruction. We hope Ev-Pano will establish a foundation to support event-guided panoramic HDR imaging and VR research community. With Ev-Pano, we further propose a novel approach that employs a weighting function-based luminance fusion to enable events to recover missing textures in LDR panoramic videos for panoramic HDR video reconstruction. We conduct extensive experiments to demonstrate the effectiveness of our approach. The results show the best performance of ours than prior arts. Meanwhile, a user study on an HDR-capable head-mounted display (Apple Vision Pro) shows feasible perceptual quality (which is closer to the HDR ground truth) of the reconstructed panoramic HDR videos. The codes and part of the dataset can be accessed via the anonymized link https://anonymous.4open.science/r/Ev-Pano-D2D2/.