Lune

ACM MM2021Top-tier venue

iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering

Liao Wang, Ziyu Wang, Pei Lin, Yuheng Jiang, Xin Suo, Minye Wu, Lan Xu, Jingyi Yu

Abstract

Generating "bullet-time" effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effects. Our iButter approach consists of a real-time preview and design stage as well as a trajectory-aware refinement stage. During preview, we propose an interactive bullet-time design approach by extending the NeRF rendering to a real-time and dynamic setting and getting rid of the tedious per-scene training. To this end, our bullet-time design stage utilizes a hybrid training set, light-weight network design and an efficient silhouette-based sampling strategy. During refinement, we introduce an efficient trajectory-aware scheme within 20 minutes, which jointly encodes the spatial, temporal consistency and semantic cues along the designed trajectory, achieving photo-realistic bullet-time viewing experience of human activities. Extensive experiments demonstrate the effectiveness of our approach for convenient interactive bullet-time design and photo-realistic human free-viewpoint video generation.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
  • NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
  • NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
  • NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object Interactions cites Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
  • VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
  • VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites Nerfies: Deformable Neural Radiance Fields
  • VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos
  • VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams cites Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
  • HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs cites MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
  • HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs cites Nerfies: Deformable Neural Radiance Fields
  • PlenOctrees for Real-time Rendering of Neural Radiance Fields cites PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
  • MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
  • Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites HumanNeRF: Efficiently Generated Human Radiance Field from Sparse Inputs
  • Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
  • Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites Nerfies: Deformable Neural Radiance Fields
  • Neural Residual Radiance Fields for Streamably Free-Viewpoint Videos cites Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time
  • Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
  • Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time cites MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
  • KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs cites PlenOctrees for Real-time Rendering of Neural Radiance Fields
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-timeLiao Wang, Jiakai Zhang, Xinhang Liu, Fuqiang Zhao et al.CVPR 2022 · 113 citations
  2. HumanNeRF: Efficiently Generated Human Radiance Field from Sparse InputsFuqiang Zhao, Wei Yang, Jiakai Zhang, Pei Lin et al.CVPR 2022 · 109 citations
  3. VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video StreamsLiao Wang, Kaixin Yao, Chengcheng Guo, Zhirui Zhang et al.CVPR 2024 · 14 citations
  4. NeuralDome: A Neural Modeling Pipeline on Multi-View Human-Object InteractionsJuze Zhang, Haimin Luo, Hongdi Yang, Xinru Xu et al.CVPR 2023
  5. Neural Residual Radiance Fields for Streamably Free-Viewpoint VideosLiao Wang, Qiang Hu, Qihan He, Ziyu Wang et al.CVPR 2023

Builds on19

  1. Nerfies: Deformable Neural Radiance FieldsKeunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz et al.ICCV 2021 · 1,442 citations
  2. PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human DigitizationShunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima et al.ICCV 2019 · 1,411 citations
  3. PlenOctrees for Real-time Rendering of Neural Radiance FieldsAlex Yu, Ruilong Li, Matthew Tancik, Hao Li et al.ICCV 2021 · 1,284 citations
  4. MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View StereoAnpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang et al.ICCV 2021 · 1,024 citations
  5. KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPsChristian Reiser, Songyou Peng, Yiyi Liao, Andreas GeigerICCV 2021 · 963 citations

Related papers

  1. Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular VideoEdgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer et al.ICCV 2021 · 617 citations
  2. Editable free-viewpoint video using a layered neural representationJiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao et al.SIGGRAPH 2021 · 80 citations
  3. Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-Aware Spatio-Temporal SamplingXinhang Liu, Yu-Wing Tai, Chi-Keung Tang, Pedro Miraldo et al.CVPR 2024
  4. NeuralHumanFVV: Real-Time Neural Volumetric Human Performance Rendering Using RGB CamerasXin Suo, Yuheng Jiang, Pei Lin, Yingliang Zhang et al.CVPR 2021
  5. FlexNeRF: Photorealistic Free-viewpoint Rendering of Moving Humans from Sparse ViewsVinoj Jayasundara, Amit Agrawal, Nicolas Heron, Abhinav Shrivastava et al.CVPR 2023
Dusk over the sea between two cliffs drawn in fine vertical lines