Peering into the Unknown: Active View Selection with Neural Uncertainty Maps for 3D Reconstruction

Imagine trying to understand the shape of a teapot by viewing it from the front-you might see the spout, but completely miss the handle. Some perspectives naturally provide more information than others. How can an AI system determine which viewpoint offers the most valuable insight for accurate and efficient 3D object reconstruction? Active view selection (AVS) for 3D reconstruction remains a fundamental challenge in computer vision. The aim is to identify the minimal set of views that yields the most accurate 3D reconstruction. Instead of learning radiance fields, like NeRF or 3D Gaussian Splatting, from a current observation and computing uncertainty for each candidate viewpoint, we introduce a novel AVS approach guided by neural uncertainty maps predicted by a lightweight feedforward deep neural network, named UPNet. UPNet takes a single input image of a 3D object and outputs a predicted uncertainty map, representing uncertainty values across all possible candidate viewpoints. By leveraging heuristics derived from observing many natural objects and their associated uncertainty patterns, we train UPNet to learn a direct mapping from viewpoint appearance to uncertainty in the underlying volumetric representations. Next, our approach aggregates all previously predicted neural uncertainty maps to suppress redundant candidate viewpoints and effectively select the most informative one. Using these selected viewpoints, we train 3D neural rendering models and evaluate the quality of novel view synthesis against other competitive AVS methods. Remarkably, despite using half of the viewpoints than the upper bound, our method achieves comparable reconstruction accuracy. In addition, it significantly reduces computational overhead during AVS, achieving up to a 400 times speedup along with over 50% reductions in CPU, RAM, and GPU usage compared to baseline methods. Notably, our approach generalizes effectively to AVS tasks involving novel object categories, without requiring any additional training. All code, models, and datasets are available at https://github. com/ZhangLab-DeepNeuroCogLab/PUN . INTRODUCTION Actively interacting with the environment to reduce uncertainty and minimize prediction errors is a fundamental capability of embodied intelligent systems Han & Zhang (2024); Friston (2010); Zhang & Xu (2024) . Some viewpoints naturally offer more informative observations than others-for example, a front view of a teapot may only reveal the spout, providing limited information, whereas a side view can expose both the handle, the spout, and detailed surface textures of the body. Active View Selection (AVS) Sequeira et al. (1996); Jia et al. (2009); Connolly (1985) ; Pito (1999) addresses this problem by selecting a minimal set of viewpoints that collectively maximize information gain. See Fig. 1(a) for an illustration of the AVS task in the context of 3D object reconstruction. AVS is critical in a range of real-world applications, including robotic control Khandelwal et al. (2023); Zhang et al. (2018b); Lv et al. (2023), search and rescue Zhang et al.