Automated Conversion of Music Videos into Lyric Videos
Jiaju Ma, Anyi Rao, Li-Yi Wei, Rubaiat Habib Kazi, Hijung Valentina Shin, Maneesh Agrawala
Abstract
Musicians and fans often produce lyric videos, a form of music videos that showcase the song’s lyrics, for their favorite songs. However, making such videos can be challenging and time-consuming as the lyrics need to be added in synchrony and visual harmony with the video. Informed by prior work and close examination of existing lyric videos, we propose a set of design guidelines to help creators make such videos. Our guidelines ensure the readability of the lyric text while maintaining a unified focus of attention. We instantiate these guidelines in a fully automated pipeline that converts an input music video into a lyric video. We demonstrate the robustness of our pipeline by generating lyric videos from a diverse range of input sources. A user study shows that lyric videos generated by our pipeline are effective in maintaining text readability and unifying the focus of attention.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
Builds on4
- TOOD: Task-aligned One-stage Object DetectionChengjian Feng, Yujie Zhong, Yu Gao, Matthew R. Scott et al.ICCV 2021 · 1,191 citations
- Objects365: A Large-Scale, High-Quality Dataset for Object DetectionShuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng et al.ICCV 2019 · 1,018 citations
- A View on the Viewer: Gaze-Adaptive Captions for VideosKuno Kurzhals, Fabian Göbel, Katrin Angerbauer, Michael Sedlmair et al.CHI 2020 · 42 citations
- A Local-to-Global Approach to Multi-Modal Movie Scene SegmentationAnyi Rao, Linning Xu, Yu Xiong, Guodong Xu et al.CVPR 2020
Related papers
- Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical ApplicationsJun Kato, Masataka GotoCHI 2023 · 4 citations
- MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song TranslationWoohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae YuEMNLP 2025
- MM-ALT: A Multimodal Automatic Lyric Transcription SystemXiangming Gu, Longshen Ou, Danielle Ong, Ye WangACM MM 2022 · 13 citations
- Unsupervised Melody-to-Lyrics GenerationYufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone et al.ACL 2023 · 6 citations
- MVPrompt: Building Music-Visual Prompts for AI Artists to Craft Music Video Mise-en-scèneChungHa Lee, Daeho Lee, Jin-Hyuk HongCHI 2025 · 5 citations
