Lune

UIST2023Top-tier venue

Automated Conversion of Music Videos into Lyric Videos

Jiaju Ma, Anyi Rao, Li-Yi Wei, Rubaiat Habib Kazi, Hijung Valentina Shin, Maneesh Agrawala

Abstract

Musicians and fans often produce lyric videos, a form of music videos that showcase the song’s lyrics, for their favorite songs. However, making such videos can be challenging and time-consuming as the lyrics need to be added in synchrony and visual harmony with the video. Informed by prior work and close examination of existing lyric videos, we propose a set of design guidelines to help creators make such videos. Our guidelines ensure the readability of the lyric text while maintaining a unified focus of attention. We instantiate these guidelines in a fully automated pipeline that converts an input music video into a lyric video. We demonstrate the robustness of our pipeline by generating lyric videos from a diverse range of input sources. A user study shows that lyric videos generated by our pipeline are effective in maintaining text readability and unifying the focus of attention.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ComVi: Context-Aware Optimized Comment Display in Video Playback cites A View on the Viewer: Gaze-Adaptive Captions for Videos
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. ScriptViz: A Visualization Tool to Aid Scriptwriting based on a Large Movie DatabaseAnyi Rao, Jean-Peïc Chou, Maneesh AgrawalaUIST 2024 · 9 citations
  2. ComVi: Context-Aware Optimized Comment Display in Video PlaybackMinsun Kim, Dawon Lee, Junyong NohCHI 2026 · 1 citation

Builds on4

  1. TOOD: Task-aligned One-stage Object DetectionChengjian Feng, Yujie Zhong, Yu Gao, Matthew R. Scott et al.ICCV 2021 · 1,191 citations
  2. Objects365: A Large-Scale, High-Quality Dataset for Object DetectionShuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng et al.ICCV 2019 · 1,018 citations
  3. A View on the Viewer: Gaze-Adaptive Captions for VideosKuno Kurzhals, Fabian Göbel, Katrin Angerbauer, Michael Sedlmair et al.CHI 2020 · 42 citations
  4. A Local-to-Global Approach to Multi-Modal Movie Scene SegmentationAnyi Rao, Linning Xu, Yu Xiong, Guodong Xu et al.CVPR 2020

Related papers

  1. Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical ApplicationsJun Kato, Masataka GotoCHI 2023 · 4 citations
  2. MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song TranslationWoohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae YuEMNLP 2025
  3. MM-ALT: A Multimodal Automatic Lyric Transcription SystemXiangming Gu, Longshen Ou, Danielle Ong, Ye WangACM MM 2022 · 13 citations
  4. Unsupervised Melody-to-Lyrics GenerationYufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone et al.ACL 2023 · 6 citations
  5. MVPrompt: Building Music-Visual Prompts for AI Artists to Craft Music Video Mise-en-scèneChungHa Lee, Daeho Lee, Jin-Hyuk HongCHI 2025 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines