Deep 3D Capture: Geometry and Reflectance From Sparse Multi-View Images

We introduce a novel learning-based method to reconstruct the high-quality geometry and complex, spatiallyvarying BRDF of an arbitrary object from a sparse set of only six images captured by wide-baseline cameras under collocated point lighting. We first estimate per-view depth maps using a deep multi-view stereo network; these depth maps are used to coarsely align the different views. We propose a novel multi-view reflectance estimation network architecture that is trained to pool features from these coarsely aligned images and predict per-view spatially-varying diffuse albedo, surface normals, specular roughness and specular albedo. Finally, we fuse and refine these per-view estimates to construct high-quality geometry and per-vertex BRDFs. We do this by jointly optimizing the latent space of our multiview reflectance network to minimize the photometric error between images rendered with our predictions and the input images. While previous state-of-the-art methods fail on such sparse acquisition setups, we demonstrate, via extensive experiments on synthetic and real data, that our method produces high-quality reconstructions that can be used to render photorealistic images.