SOMSI: Spherical Novel View Synthesis with Soft Occlusion Multi-Sphere Images

Spherical novel view synthesis (SNVS) is the task of estimating 360 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">○</sup> views at dynamic novel views given a set of 360 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">○</sup> input views. Prior arts learn multi-sphere image (MSI) representations that enable fast rendering times but are only limited to modelling low-dimensional color values. Modelling high-dimensional appearance features in MSI can result in better view synthesis, but it is not feasible to represent high-dimensional features in a large number (> 64) of MSI spheres. We propose a novel MSI representation called Soft Occlusion MSI (SOMSI) that enables modelling high-dimensional appearance features in MSI while retaining the fast rendering times of a standard MSI. Our key insight is to model appearance features in a smaller set (e.g. 3) of occlusion levels instead of larger number of MSI levels. Experiments on both synthetic and real-world scenes demonstrate that using SOMSI can provide a good balance between accuracy and run-time. SOMSI can produce considerably better results compared to MSI based MODS [1], while having similar fast rendering time. SOMSI view synthesis quality is on-par with state-of-the-art NeRF [24] like model while being 2 orders of magnitude faster. For code, additional results and data, please visit https://tedyhabtegebrial.github.io/somsi.