Lune

EMNLP Findings2023

Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics

Chengxi Li, Kai Fan, Jiajun Bu, Boxing Chen, Zhongqiang Huang, Zhi Yu

Abstract

Song translation requires both translation of lyrics and alignment of music notes so that the resulting verse can be sung to the accompanying melody, which is a challenging problem that has attracted some interests in different aspects of the translation process. In this paper, we propose Lyrics-Melody Translation with Adaptive Grouping (LTAG), a holistic solution to automatic song translation by jointly modeling lyrics translation and lyrics-melody alignment. It is a novel encoder-decoder framework that can simultaneously translate the source lyrics and determine the number of aligned notes at each decoding step through an adaptive note grouping module. To address data scarcity, we commissioned a small amount of training data annotated specifically for this task and used large amounts of augmented data through back-translation. Experiments conducted on an English-Chinese song translation data set show the effectiveness of our model in both automatic and human evaluations. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Learning the Beauty in Songs: Neural Singing Voice Beautifier cites DeepSinger: Singing Voice Synthesis with Data Mined From the Web
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song TranslationWoohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae YuEMNLP 2025

Builds on11

  1. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  2. SongMASS: Automatic Song Writing with Pre-training and Alignment ConstraintZhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren et al.AAAI 2021 · 84 citations
  3. DeepSinger: Singing Voice Synthesis with Data Mined From the WebYi Ren, Xu Tan, Tao Qin, Jian Luan et al.KDD 2020 · 72 citations
  4. Controllable Generation from Pre-trained Language Models via Inverse PromptingXu Zou, Da Yin, Qingyang Zhong, Hongxia Yang et al.KDD 2021 · 29 citations
  5. Learning the Beauty in Songs: Neural Singing Voice BeautifierJinglin Liu, Chengxi Li, Yi Ren, Zhiying Zhu et al.ACL 2022 · 25 citations

Related papers

  1. SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-TrainingJiaxing Yu, Xinda Wu, Yunfei Xu, Tieyao Zhang et al.AAAI 2025 · 2 citations
  2. Automatic Song Translation for Tonal LanguagesFenfei Guo, Chen Zhang, Zhirui Zhang, Qixin He et al.ACL Findings 2022 · 6 citations
  3. Unsupervised Melody-to-Lyrics GenerationYufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone et al.ACL 2023 · 6 citations
  4. Songs Across Borders: Singable and Controllable Neural Lyric TranslationLongshen Ou, Xichu Ma, Min-Yen Kan, Ye WangACL 2023 · 3 citations
  5. ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody RelationshipsChen Zhang, LuChin Chang, Songruoyao Wu, Xu Tan et al.ACM MM 2022 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines