Lune

ACM MM2021Top-tier venue

Dual Learning Music Composition and Dance Choreography

Shuang Wu, Zhenguang Liu, Shijian Lu, Li Cheng

Abstract

Music and dance have always co-existed as pillars of human activities, contributing immensely to the cultural, social, and entertainment functions in virtually all societies. Notwithstanding the gradual systematization of music and dance into two independent disciplines, their intimate connection is undeniable and one art-form often appears incomplete without the other. Recent research works have studied generative models for dance sequences conditioned on music. The dual task of composing music for given dances, however, has been largely overlooked. In this paper, we propose a novel extension, where we jointly model both tasks in a dual learning approach. To leverage the duality of the two modalities, we introduce an optimal transport objective to align feature embeddings, as well as a cycle consistency loss to foster overall consistency. Experimental results demonstrate that our dual learning framework improves individual task performance, delivering generated music compositions and dance choreographs that are realistic and faithful to the conditioned inputs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • GroupDancer: Music to Multi-People Dance Synthesis with Style Collaboration cites ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
  • GroupDancer: Music to Multi-People Dance Synthesis with Style Collaboration cites Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
  • GroupDancer: Music to Multi-People Dance Synthesis with Style Collaboration cites AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
  • Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning cites ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
  • AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ cites ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
  • AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ cites Self-supervised Dance Video Synthesis Conditioned on Music
  • AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ cites Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
  • DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis cites ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
  • DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis cites GroupDancer: Music to Multi-People Dance Synthesis with Style Collaboration
  • DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis cites Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
  • DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis cites AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
  • DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesis cites DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance
  • POPDG: Popular 3D Dance Generation with PopDanceSet cites ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
  • POPDG: Popular 3D Dance Generation with PopDanceSet cites Self-supervised Dance Video Synthesis Conditioned on Music
  • POPDG: Popular 3D Dance Generation with PopDanceSet cites Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
  • POPDG: Popular 3D Dance Generation with PopDanceSet cites AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
  • DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance cites ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit
  • DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance cites GroupDancer: Music to Multi-People Dance Synthesis with Style Collaboration
  • DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance cites Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning
  • DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance cites AI Choreographer: Music Conditioned 3D Dance Generation with AIST++
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. GroupDancer: Music to Multi-People Dance Synthesis with Style CollaborationZixuan Wang, Jia Jia, Haozhe Wu, Junliang Xing et al.ACM MM 2022 · 17 citations
  2. POPDG: Popular 3D Dance Generation with PopDanceSetZhenye Luo, Min Ren, Xuecai Hu, Yongzhen Huang et al.CVPR 2024 · 11 citations
  3. DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera SynthesisZixuan Wang, Jiayi Li, Xiaoyu Qin, Shikun Sun et al.ACM MM 2024 · 5 citations
  4. DanceCamera3D: 3D Camera Movement Synthesis with Music and DanceZixuan Wang, Jia Jia, Shikun Sun, Haozhe Wu et al.CVPR 2024 · 5 citations

Builds on5

  1. AI Choreographer: Music Conditioned 3D Dance Generation with AIST++Ruilong Li, Shan Yang, David A. Ross, Angjoo KanazawaICCV 2021 · 701 citations
  2. Pop Music Transformer: Beat-based Modeling and Generation of Expressive Pop Piano CompositionsYu-Siang Huang, Yi-Hsuan YangACM MM 2020 · 265 citations
  3. Dance Revolution: Long-Term Dance Generation with Music via Curriculum LearningRuozi Huang, Huang Hu, Wei Wu, Kei Sawada et al.ICLR 2021 · 147 citations
  4. Self-supervised Dance Video Synthesis Conditioned on MusicXuanchi Ren, Haoran Li, Zijian Huang, Qifeng ChenACM MM 2020 · 68 citations
  5. ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action UnitZijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu et al.ACM MM 2020 · 59 citations

Related papers

  1. ChoreoGraph: Music-conditioned Automatic Dance Choreography over a Style and Tempo Consistent Dynamic GraphHo Yin Au, Jie Chen, Junkun Jiang, Yike GuoACM MM 2022 · 29 citations
  2. TM2D: Bimodality Driven 3D Dance Generation via Music-Text IntegrationKehong Gong, Dongze Lian, Heng Chang, Chuan Guo et al.ICCV 2023 · 103 citations
  3. MoMu-Diffusion: On Learning Long-Term Motion-Music Synchronization and CorrespondenceFuming You, Minghui Fang, Li Tang, Rongjie Huang et al.NeurIPS 2024 · 8 citations
  4. Mining and Applying Composition Knowledge of Dance Moves for Style-Concentrated Dance GenerationXinjian Zhang, Su Yang, Yi Xu, Weishan Zhang et al.AAAI 2023 · 4 citations
  5. Enhancing Dance-to-Music Generation via Negative Conditioning Latent Diffusion ModelChangchang Sun, Gaowen Liu, Charles Fleming, Yan YanCVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines