Lune

Abstract

Our goal is to learn a deep network that, given a small number of images of an object of a given category, reconstructs it in 3D. While several recent works have obtained analogous results using synthetic data or assuming the avail-ability of 2D primitives such as keypoints, we are interested in working with challenging real data and with no manual annotations. We thus focus on learning a model from multiple views of a large collection of object instances. We contribute with a new large dataset of object centric videos suitable for training and benchmarking this class of models. We show that existing techniques leveraging meshes, voxels, or implicit surfaces, which work well for reconstructing isolated objects, fail on this challenging data. Finally, we propose a new neural network design, called warp-conditioned ray embedding (WCR), which significantly improves reconstruction while obtaining a detailed implicit representation of the object surface and texture, also compensating for the noise in the initial SfM reconstruction that bootstrapped the learning process. Our evaluation demonstrates performance improvements over several deep monocular reconstruction baselines on existing benchmarks and on our novel dataset. For additional material please visit: https://henzler.github.io/publication/unsupervised_videos/.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

33 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization cites Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction
  • ReconFusion: 3D Reconstruction with Diffusion Priors cites Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data
  • Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data cites Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction
  • GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct Supervision cites Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction
  • Neural Sparse Voxel Fields cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction cites PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows
  • Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction cites GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
  • Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction cites Neural Sparse Voxel Fields
  • Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction cites Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers33

  1. One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape OptimizationMinghua Liu, Chao Xu, Haian Jin, Linghao Chen et al.NeurIPS 2023 · 755 citations
  2. Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category ReconstructionJeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone et al.ICCV 2021 · 686 citations
  3. ReconFusion: 3D Reconstruction with Diffusion PriorsRundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park et al.CVPR 2024 · 137 citations
  4. Diffusion with Forward Models: Solving Stochastic Inverse Problems Without Direct SupervisionAyush Tewari, Tianwei Yin, George Cazenavette, Semon Rezchikov et al.NeurIPS 2023 · 131 citations
  5. Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D DataStanislaw Szymanowicz, Christian Rupprecht, Andrea VedaldiICCV 2023 · 130 citations

Builds on19

  1. Neural Sparse Voxel FieldsLingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua et al.NeurIPS 2020 · 1,535 citations
  2. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human DigitizationShunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima et al.ICCV 2019 · 1,411 citations
  3. Multiview Neural Surface Reconstruction by Disentangling Geometry and AppearanceLior Yariv, Yoni Kasten, Dror Moran, Meirav Galun et al.NeurIPS 2020 · 1,010 citations
  4. GRAF: Generative Radiance Fields for 3D-Aware Image SynthesisKatja Schwarz, Yiyi Liao, Michael Niemeyer, Andreas GeigerNeurIPS 2020 · 1,001 citations
  5. PointFlow: 3D Point Cloud Generation With Continuous Normalizing FlowsGuandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu et al.ICCV 2019 · 794 citations

Related papers

  1. Online Adaptation for Consistent Mesh Reconstruction in the WildXueting Li, Sifei Liu, Shalini De Mello, Kihwan Kim et al.NeurIPS 2020 · 62 citations
  2. Object-Centric Learning for Real-World Videos by Predicting Temporal Feature SimilaritiesAndrii Zadaianchuk, Maximilian Seitzer, Georg MartiusNeurIPS 2023 · 104 citations
  3. Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New DatasetYanjie Ze, Xiaolong WangNeurIPS 2022 · 104 citations
  4. Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown CamerasAriel Gordon, Hanhan Li, Rico Jonschkowski, Anelia AngelovaICCV 2019 · 397 citations
  5. Common3D: Self-Supervised Learning of 3D Morphable Models for Common Objects in Neural Feature SpaceLeonhard Sommer, Olaf Dünkel, Christian Theobalt, Adam KortylewskiCVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines