iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering
Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, Jingyi Yu
Abstract
Generating "bullet-time" effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects. Our iButter approach consists of a real-time preview and design stage as well as a trajectory-aware refinement stage. During preview, we propose an interactive bullet-time design approach by extending the NeRF rendering to a real-time and dynamic setting and getting rid of the tedious per-scene training. To this end, our bullet-time design stage utilizes a hybrid training set, light-weight network design and an efficient silhouette-based sampling strategy. During refinement, we introduce an efficient trajectory-aware scheme within 20 minutes, which jointly encodes the spatial, temporal consistency and semantic cues along the designed trajectory, achieving photo-realistic bullet-time viewing experience of human activities. Extensive experiments demonstrate the effectiveness of our approach for convenient interactive bullet-time design and photo-realistic human free-viewpoint video generation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
- NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
- NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
- VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
- VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites Nerfies: Deformable Neural Radiance Fields
- VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
- VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
- HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs cites MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs cites Nerfies: Deformable Neural Radiance Fields
- PlenOctrees for Real-time Rendering of Neural Radiance Fields cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
- Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
- Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
- Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites Nerfies: Deformable Neural Radiance Fields
- Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
- Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
- Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time cites MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
Cited by top-tier papers6
- Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-timeLiao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao et al.CVPR 2022 · 113 citations
- HumanNeRF: Efficiently Generated Human Radiance Field from Sparse InputsFuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin et al.CVPR 2022 · 109 citations
- VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video StreamsLiao Wang, Kaixin Yao, Chengcheng Guo, Zhirui Zhang et al.CVPR 2024 · 14 citations
- NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object InteractionsJuze Zhang, Haimin Luo, Hongdi Yang, Xinru Xu et al.CVPR 2023
- Neural Residual Radiance Fields for Streamably Free-Viewpoint VideosLiao Wang, Qiang Hu, Qihan He, Ziyu Wang et al.CVPR 2023
Builds on19
- Nerfies: Deformable Neural Radiance FieldsKeunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz et al.ICCV 2021 · 1,442 citations
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human DigitizationShunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima et al.ICCV 2019 · 1,411 citations
- PlenOctrees for Real-time Rendering of Neural Radiance FieldsAlex Yu, Ruilong Li, Matthew Tancik, Hao Li et al.ICCV 2021 · 1,284 citations
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View StereoAnpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang et al.ICCV 2021 · 1,024 citations
- KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPsChristian Reiser, Songyou Peng, Yiyi Liao, Andreas GeigerICCV 2021 · 963 citations
Related papers
- Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular VideoEdgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer et al.ICCV 2021 · 617 citations
- Editable free-viewpoint video using a layered neural representationJiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao et al.SIGGRAPH 2021 · 80 citations
- Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-Aware Spatio-Temporal SamplingXinhang Liu, Yu-Wing Tai, Chi-Keung Tang, Pedro Miraldo et al.CVPR 2024
- NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering Using RGB CamerasXin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang et al.CVPR 2021
- FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from Sparse ViewsVinoj Jayasundara, Amit Agrawal, Nicolas Heron, Abhinav Shrivastava et al.CVPR 2023
