NeRM: Learning Neural Representations for High-Framerate Human Motion Synthesis

Generating realistic human motions with high framerate is an underexplored task, due to the varied framerates of training data, huge memory burden brought by high framerates and slow sampling speed of generative models. Recent advances make a compromise for training by downsampling high-framerate details away and discarding low-framerate samples, which suffer from severe information loss and restricted-framerate generation. In this paper, we found that the recent emerging paradigm of Implicit Neural Representations (INRs) that encode a signal into a continuous function can effectively tackle this challenging problem. To this end, we introduce NeRM, a generative model capable of taking advantage of variedsize data and capturing variational distribution of motions for high-framerate motion synthesis. By optimizing latent representation and an auto-decoder conditioned on temporal coordinates, NeRM learns neural representations for sampled motion clips that ingeniously avoid explicit modeling of raw varied-size motions. This expressive latent representation is then used to learn a diffusion model that enables both unconditional and conditional generation of human motions. We show that NeRM not only achieves competitive results with state-of-the-art methods, but also be capable of generating arbitrary-framerate motions. Moreover, it can remain memory-friendly yet highly efficient even when generating highframerate motions.