Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation
Ziyang Xie, Zhizheng Liu, Zhenghao Peng, Wayne Wu, Bolei Zhou
2025Year
9Top-tier citations
Abstract
Figure 1. Vid2Sim converts monocular video captured by a hand-held camera into realistic and interactive 3D simulation environments. It facilitates RL training of navigation agents in digital twins of urban scenes and provides realistic observations like RGB and depth to reduce the sim-to-real gap. The pink mobile robot in the image is a food delivery bot that avoids collisions with pedestrians and obstacles.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext b4c57f14-aa79-40e3-9f1a-8ac9fe78dbf8Cited by top-tier papers9
- SimScale: Learning to Drive via Real-World Simulation at ScaleHaochen Tian, Tianyu Li, Haochen Liu, Jiazhi Yang et al.CVPR 2026 · 40 citations
- From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement LearningHonglin He, Yukai Ma, Brad Squicciarini, Wayne Wu et al.ICLR 2026 · 19 citations
- Model-Based Policy Adaptation for Closed-Loop End-to-end Autonomous DrivingHaohong Lin, Yunzhi Zhang, Wenhao Ding, Jiajun Wu et al.NeurIPS 2025 · 11 citations
- CapNav: Benchmarking Vision Language Models on Capability-conditioned Indoor NavigationXia Su, Ruiqi Chen, Benlin Liu, Jingwei Ma et al.CVPR 2026 · 8 citations
- UrbanVerse: Scaling Urban Simulation by Watching City-Tour VideosMingxuan Liu, Honglin He, Elisa Ricci, Wayne Wu et al.ICLR 2026 · 8 citations
Builds on24
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
- 3D Gaussian Splatting for Real-Time Radiance Field RenderingBernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George DrettakisSIGGRAPH 2023 · 5,687 citations
- Instant neural graphics primitives with a multiresolution hash encodingThomas Müller, Alex Evans, Christoph Schied, Alexander KellerSIGGRAPH 2022 · 4,089 citations
- NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view ReconstructionPeng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt et al.NeurIPS 2021 · 2,500 citations
Related papers
- Neural Lighting Simulation for Urban ScenesAva Pun, Gary Sun, Jingkang Wang, Yun Chen et al.NeurIPS 2023 · 11 citations
- Wanderland: Geometrically Grounded Simulation for Open-World Embodied AIXinhao Liu, Jiaqi Li, Youming Deng, Ruxin Chen et al.CVPR 2026 · 5 citations
- Learning 3D Particle-based Simulators from RGB-D VideosWilliam F. Whitney, Tatiana Lopez-Guevara, Tobias Pfaff, Yulia Rubanova et al.ICLR 2024 · 16 citations
- No RL, No Simulation: Learning to Navigate without NavigatingMeera Hahn, Devendra Singh Chaplot, Shubham Tulsiani, Mustafa Mukadam et al.NeurIPS 2021 · 98 citations
- Contact-guided Real2Sim from Monocular Video with Planar Scene PrimitivesZihan Wang, Jiashun Wang, Jeff Tan, Yiwen Zhao et al.ICLR 2026
