Lune

CHI2023Top-tier venue

Lyric App Framework: A Web-based Framework for Developing Interactive Lyric-driven Musical Applications

Jun Kato, Masataka Goto

Abstract

Lyric videos have become a popular medium to convey lyrical content to listeners, but they present the same content whenever they are played and cannot adapt to listeners’ preferences. Lyric apps, as we name them, are a new form of lyric-driven visual art that can render different lyrical content depending on user interaction and address the limitations of static media. To open up this novel design space for programmers and musicians, we present Lyric App Framework, a web-based framework for building interactive graphical applications that play musical pieces and show lyrics synchronized with playback. We designed the framework to provide a streamlined development experience for building production-ready lyric apps with creative coding libraries of choice. We held programming contests twice and collected 52 examples of lyric apps, enabling us to reveal eight representative categories, confirm the framework’s effectiveness, and report lessons learned.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on2

  1. Crosscast: Adding Visuals to Audio Travel PodcastsHaijun Xia, Jennifer Jacobs, Maneesh AgrawalaUIST 2020 · 44 citations
  2. Learning From Music to Visual Storytelling of Shots: A Deep Interactive Learning MechanismJen-Chun Lin, Wen-Li Wei, Yen-Yu Lin, Tyng-Luh Liu et al.ACM MM 2020 · 3 citations

Related papers

  1. Automated Conversion of Music Videos into Lyric VideosJiaju Ma, Anyi Rao, Li-Yi Wei, Rubaiat Habib Kazi et al.UIST 2023 · 7 citations
  2. Unsupervised Melody-to-Lyrics GenerationYufei Tian, Anjali Narayan-Chen, Shereen Oraby, Alessandra Cervone et al.ACL 2023 · 6 citations
  3. ToneCraft: Cantonese Lyrics Generation with Harmony of Tones and PitchesJunyu Cheng, Chang Pan, Shuangyin LiEMNLP 2025
  4. UniLG: A Unified Structure-aware Framework for Lyrics GenerationTao Qian, Fan Lou, Jiatong Shi, Yuning Wu et al.ACL 2023 · 2 citations
  5. µCap: Instrumental Music Captions for Deaf and Hard-of-Hearing IndividualsSooyeon Ahn, In-Chang Baek, Kyung-Joong Kim, Khai N. Truong et al.CHI 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines