Lune

AAAI2023Top-tier venue

Multi-Scale Control Signal-Aware Transformer for Motion Synthesis without Phase

Lintao Wang, Kun Hu, Lei Bai, Yu Ding, Wanli Ouyang, Zhiyong Wang

Abstract

Synthesizing controllable motion for a character using deep learning has been a promising approach due to its potential to learn a compact model without laborious feature engineering. To produce dynamic motion from weak control signals such as desired paths, existing methods often require auxiliary information such as phases for alleviating motion ambiguity, which limits their generalisation capability. As past poses often contain useful auxiliary hints, in this paper, we propose a task-agnostic deep learning method, namely Multi-scale Control Signal-aware Transformer (MCS-T), with an attention based encoder-decoder architecture to discover the auxiliary information implicitly for synthesizing controllable motion without explicitly requiring auxiliary information such as phase. Specifically, an encoder is devised to adaptively formulate the motion patterns of a character's past poses with multi-scale skeletons, and a decoder driven by control signals to further synthesize and predict the character's state by paying context-specialised attention to the encoded past motion patterns. As a result, it helps alleviate the issues of low responsiveness and slow transition which often happen in conventional methods not using auxiliary information. Both qualitative and quantitative experimental results on an existing biped locomotion dataset, which involves diverse types of motion transitions, demonstrate the effectiveness of our method. In particular, MCS-T is able to successfully generate motions comparable to those generated by the methods using auxiliary information.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ViViT: A Video Vision Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Action-Conditioned 3D Human Motion Synthesis with Transformer VAE cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • 3D Human Pose Estimation with Spatial and Temporal Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • PUMPS: Skeleton-Agnostic Point-Based Universal Motion Pre-Training for Synthesis in Human Motion Tasks cites Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
  • PUMPS: Skeleton-Agnostic Point-Based Universal Motion Pre-Training for Synthesis in Human Motion Tasks cites 3D Human Pose Estimation with Spatial and Temporal Transformers
  • PUMPS: Skeleton-Agnostic Point-Based Universal Motion Pre-Training for Synthesis in Human Motion Tasks cites Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion Interpolation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. PUMPS: Skeleton-Agnostic Point-Based Universal Motion Pre-Training for Synthesis in Human Motion TasksClinton Ansun Mo, Kun Hu, Chengjiang Long, Dong Yuan et al.ICCV 2025 · 2 citations
  2. Continuous Intermediate Token Learning with Implicit Motion Manifold for Keyframe Based Motion InterpolationClinton Ansun Mo, Kun Hu, Chengjiang Long, Zhiyong WangCVPR 2023

Builds on15

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. ViViT: A Video Vision TransformerAnurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun et al.ICCV 2021 · 2,947 citations
  3. Action-Conditioned 3D Human Motion Synthesis with Transformer VAEMathis Petrovich, Michael J. Black, Gül VarolICCV 2021 · 672 citations
  4. 3D Human Pose Estimation with Spatial and Temporal TransformersCe Zheng, Sijie Zhu, Matías Mendieta, Taojiannan Yang et al.ICCV 2021 · 648 citations
  5. Character controllers using motion VAEsHung Yu Ling, Fabio Zinno, George Cheng, Michiel van de PanneSIGGRAPH 2020 · 261 citations

Related papers

  1. Motion Synthesis with Sparse and Flexible Keyjoint ControlInwoo Hwang, Jinseok Bae, Donggeun Lim, Young Min KimICCV 2025 · 2 citations
  2. Autoregressive Stylized Motion Synthesis With Generative FlowYu-Hui Wen, Zhipeng Yang, Hongbo Fu, Lin Gao et al.CVPR 2021
  3. Neural Marionette: Unsupervised Learning of Motion Skeleton and Latent Dynamics from Volumetric VideoJinseok Bae, Hojun Jang, Cheol-Hui Min, Hyungun Choi et al.AAAI 2022 · 6 citations
  4. Dynamic Future Net: Diversified Human Motion GenerationWenheng Chen, He Wang, Yi Yuan, Tianjia Shao et al.ACM MM 2020 · 20 citations
  5. DeepPhase: periodic autoencoders for learning motion phase manifoldsSebastian Starke, Ian Mason, Taku KomuraSIGGRAPH 2022 · 142 citations
Dusk over the sea between two cliffs drawn in fine vertical lines