EMNLP Findings2023
Translate the Beauty in Songs: Jointly Learning to Align Melody and Translate Lyrics
Chengxi Li, Kai Fan, Jiajun Bu, Boxing Chen, Zhongqiang Huang, Zhi Yu
Abstract
Song translation requires both translation of lyrics and alignment of music notes so that the resulting verse can be sung to the accompanying melody, which is a challenging problem that has attracted some interests in different aspects of the translation process. In this paper, we propose Lyrics-Melody Translation with Adaptive Grouping (LTAG), a holistic solution to automatic song translation by jointly modeling lyrics translation and lyrics-melody alignment. It is a novel encoder-decoder framework that can simultaneously translate the source lyrics and determine the number of aligned notes at each decoding step through an adaptive note grouping module. To address data scarcity, we commissioned a small amount of training data annotated specifically for this task and used large amounts of augmented data through back-translation. Experiments conducted on an English-Chinese song translation data set show the effectiveness of our model in both automatic and human evaluations. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on11
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- SongMASS: Automatic Song Writing with Pre-training and Alignment ConstraintZhonghao Sheng, Kaitao Song, Xu Tan, Yi Ren et al.AAAI 2021 · 84 citations
- DeepSinger: Singing Voice Synthesis with Data Mined From the WebYi Ren, Xu Tan, Tao Qin, Jian Luan et al.KDD 2020 · 72 citations
- Controllable Generation from Pre-trained Language Models via Inverse PromptingXu Zou, Da Yin, Qingyang Zhong, Hongxia Yang et al.KDD 2021 · 29 citations
- Learning the Beauty in Songs: Neural Singing Voice BeautifierJinglin Liu, Chengxi Li, Yi Ren, Zhiying Zhu et al.ACL 2022 · 25 citations
Related papers
- SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-TrainingJiaxing Yu, Xinda Wu, Yunfei Xu, Tieyao Zhang et al.AAAI 2025 · 2 citations
- Automatic Song Translation for Tonal LanguagesFenfei Guo, Chen Zhang, Zhirui Zhang, Qixin He et al.ACL Findings 2022 · 6 citations
- Unsupervised Melody-to-Lyrics GenerationYufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone et al.ACL 2023 · 6 citations
- Songs Across Borders: Singable and Controllable Neural Lyric TranslationLongshen Ou, Xichu Ma, Min-Yen Kan, Ye WangACL 2023 · 3 citations
- ReLyMe: Improving Lyric-to-Melody Generation by Incorporating Lyric-Melody RelationshipsChen Zhang, LuChin Chang, Songruoyao Wu, Xu Tan et al.ACM MM 2022 · 13 citations
