Lune

SIGGRAPH2025Top-tier venue

Photoreal Scene Reconstruction from an Egocentric Device

Zhaoyang Lv, Maurizio Monge, Ka Chen, Yufeng Zhu, Michael Goesele, Jakob J. Engel, Zhao Dong, Richard A. Newcombe

Abstract

In this paper, we investigate the challenges associated with using egocentric devices to photorealistic reconstruct the scene in high dynamic range. Existing methodologies typically assume using frame-rate 6DoF pose estimated from the device’s visual-inertial odometry system, which may neglect crucial details necessary for pixel-accurate reconstruction. This study presents two significant findings. Firstly, in contrast to mainstream work treating RGB camera as global shutter frame-rate camera, we emphasize the importance of employing visual-inertial bundle adjustment (VIBA) to calibrate the precise timestamps and movement of the rolling shutter RGB sensing camera in a high frequency trajectory format, which ensures an accurate calibration of the physical properties of the rolling-shutter camera. Secondly, we incorporate a physical image formation model based into Gaussian Splatting, which effectively addresses the sensor characteristics, including the rolling-shutter effect of RGB cameras and the dynamic ranges measured by sensors. Our proposed formulation is applicable to the widely-used variants of Gaussian Splats representation. We conduct a comprehensive evaluation of our pipeline using the open-source Project Aria device under diverse indoor and outdoor lighting conditions, and further validate it on a Meta Quest3 device. Across all experiments, we observe a consistent visual enhancement of +1 dB in PSNR by incorporating VIBA, with an additional +1 dB achieved through our proposed image formation model. Our complete implementation, evaluation datasets, and recording profile are available at https://www.projectaria.com/photoreal-reconstruction/

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • 3DGEER: 3D Gaussian Rendering Made Exact and Efficient for Generic Cameras cites 2D Gaussian Splatting for Geometrically Accurate Radiance Fields
  • 3DGEER: 3D Gaussian Rendering Made Exact and Efficient for Generic Cameras cites 3D Gaussian Splatting for Real-Time Radiance Field Rendering
  • 2D Gaussian Splatting for Geometrically Accurate Radiance Fields cites Instant neural graphics primitives with a multiresolution hash encoding
  • 2D Gaussian Splatting for Geometrically Accurate Radiance Fields cites Plenoxels: Radiance Fields without Neural Networks
  • 3D Gaussian Splatting for Real-Time Radiance Field Rendering cites Instant neural graphics primitives with a multiresolution hash encoding
  • 3D Gaussian Splatting for Real-Time Radiance Field Rendering cites Plenoxels: Radiance Fields without Neural Networks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. 3DGEER: 3D Gaussian Rendering Made Exact and Efficient for Generic CamerasZixun Huang, Cho-Ying Wu, Yuliang Guo, Xinyu Huang et al.ICLR 2026 · 9 citations

Builds on12

  1. 3D Gaussian Splatting for Real-Time Radiance Field RenderingBernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George DrettakisSIGGRAPH 2023 · 5,687 citations
  2. Instant neural graphics primitives with a multiresolution hash encodingThomas Müller, Alex Evans, Christoph Schied, Alexander KellerSIGGRAPH 2022 · 4,089 citations
  3. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance FieldsJonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan et al.CVPR 2022 · 1,603 citations
  4. Plenoxels: Radiance Fields without Neural NetworksSara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen et al.CVPR 2022 · 1,237 citations
  5. 2D Gaussian Splatting for Geometrically Accurate Radiance FieldsBinbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger et al.SIGGRAPH 2024 · 660 citations

Related papers

  1. Casual3DHDR: High Dynamic Range 3D Gaussian Splatting from Casually Captured VideosShucheng Gong, Lingzhe Zhao, Wenpu Li, Hong Xie et al.ACM MM 2025 · 2 citations
  2. MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D ReconstructionXiaohao Xu, Feng Xue, Shibo Zhao, Yike Pan et al.CVPR 2025
  3. EvHDR-GS: Event-guided HDR Video Reconstruction with 3D Gaussian SplattingZehao Chen, Zhan Lu, De Ma, Huajin Tang et al.AAAI 2025 · 6 citations
  4. 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian SplattingQi Wu, Janick Martinez Esturo, Ashkan Mirzaei, Nicolas Moënne-Loccoz et al.CVPR 2025
  5. Event-guided Frame Interpolation and Dynamic Range Expansion of Single Rolling Shutter ImageGuixu Lin, Jin Han, Mingdeng Cao, Zhihang Zhong et al.ACM MM 2023 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines