HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion

Novel-view Synthesis with HumanRF Fig. 1. We introduce a novel multi-view dataset of humans in motion captured with a rig of 160 cameras, recording footage of 12MP each (left image). From an input multi-view recording of a specific person, our HumanRF method reconstructs a spatio-temporal radiance field which captures appearance and motion of the actor. From this representation, we can then synthesize highly-realistic images from unseen, novel view points (right images).

Representing human performance at high-fidelity is an essential building block in diverse applications, such as film production, computer games or videoconferencing. To close the gap to production-level quality, we introduce HumanRF 1 , a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints. Our novel representation acts as a dynamic video encoding that captures fine details at high compression rates by factorizing space-time into a temporal matrix-vector decomposition. This allows us to obtain temporally coherent reconstructions of human actors for long sequences, while representing high-resolution details even in the context of challenging motion. While most research focuses on synthesizing at resolutions of 4MP or lower, we address the challenge of operating at 1 Project website: synthesiaresearch.github.io/humanrf