Differentiable Stereopsis: Meshes from multiple views using differentiable rendering
Abstract
We propose Differentiable Stereopsis, a multi-view stereo approach that reconstructs shape and texture from few input views and noisy cameras. We pair traditional stereopsis and modern differentiable rendering to build an end-to-end model which predicts textured 3D meshes of objects with varying topologies and shape. We frame stereopsis as an optimization problem and simultaneously update shape and cameras via simple gradient descent. We run an extensive quantitative analysis and compare to traditional multi-view stereo techniques and state-of-the-art learning based methods. We show compelling reconstructions on challenging real-world scenes and for an abundance of object types with complex shape, topology and texture. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Project webpage: https://shubham-goel.github.io/ds/
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Shelf-Supervised Mesh Prediction in the Wild cites Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
- Shelf-Supervised Mesh Prediction in the Wild cites Articulation-Aware Canonical Surface Mapping
- Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials cites Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
- Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials cites Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning
- Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materials cites Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
- Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives cites Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
- Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives cites Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
- Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance cites Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision
Cited by top-tier papers12
- Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single ImageKailu Wu, Fangfu Liu, Zhihan Cai, Runjie Yan et al.NeurIPS 2024 · 185 citations
- Cameras as Rays: Pose Estimation via Ray DiffusionJason Y. Zhang, Amy Lin, Moneish Kumar, Tzu-Hsuan Yang et al.ICLR 2024 · 126 citations
- Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR MaterialsYawar Siddiqui, Tom Monnier, Filippos Kokkinos, Mahendra Kariya et al.NeurIPS 2024 · 89 citations
- Differentiable Blocks World: Qualitative 3D Decomposition by Rendering PrimitivesTom Monnier, Jake Austin, Angjoo Kanazawa, Alexei A. Efros et al.NeurIPS 2023 · 50 citations
- HollowNeRF: Pruning Hashgrid-Based NeRFs with Trainable Collision MitigationXiufeng Xie, Riccardo Gherardi, Zhihong Pan, Stephen HuangICCV 2023 · 21 citations
Builds on5
- Multiview Neural Surface Reconstruction by Disentangling Geometry and AppearanceLior Yariv, Yoni Kasten, Dror Moran, Meirav Galun et al.NeurIPS 2020 · 1,010 citations
- Soft Rasterizer: A Differentiable Renderer for Image-Based 3D ReasoningShichen Liu, Weikai Chen, Tianye Li, Hao LiICCV 2019 · 789 citations
- Shelf-Supervised Mesh Prediction in the WildYufei Ye, Shubham Tulsiani, Abhinav GuptaCVPR 2021
- Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D SupervisionMichael Niemeyer, Lars M. Mescheder, Michael Oechsle, Andreas GeigerCVPR 2020
- Articulation-Aware Canonical Surface MappingNilesh Kulkarni, Abhinav Gupta, David F. Fouhey, Shubham TulsianiCVPR 2020
Related papers
- TMO: Textured Mesh Acquisition of Objects with a Mobile Device by using Differentiable RenderingJaehoon Choi, Dongki Jung, Taejae Lee, Sangwook Kim et al.CVPR 2023
- Multi-View Reconstruction Using Signed Ray Distance Functions (SRDF)Pierre Zins, Yuanlu Xu, Edmond Boyer, Stefanie Wuhrer et al.CVPR 2023
- HelixSurf: A Robust and Efficient Neural Implicit Surface Learning of Indoor Scenes with Iterative Intertwined RegularizationZhihao Liang, Zhangjin Huang, Changxing Ding, Kui JiaCVPR 2023
- SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape OptimizationYue Jiang, Dantong Ji, Zhizhong Han, Matthias ZwickerCVPR 2020
- DMesh: A Differentiable Mesh RepresentationSanghyun Son, Matheus Gadelha, Yang Zhou, Zexiang Xu et al.NeurIPS 2024 · 11 citations
