ELMI: Interactive and Intelligent Sign Language Translation of Lyrics for Song Signing
Suhyeon Yoo, Khai N. Truong, Young-Ho Kim
Abstract
d/Deaf and hearing song-signers have become prevalent across video-sharing platforms, but translating songs into sign language remains cumbersome and inaccessible. Our formative study revealed the challenges song-signers face, including semantic, syntactic, expressive, and rhythmic considerations in translations. We present ELMI, an accessible song-signing tool that assists in translating lyrics into sign language. ELMI enables users to edit glosses line-by-line, with real-time synced lyric and music video snippets. Users can also chat with a large language model-driven AI to discuss meaning, glossing, emoting, and timing. Through an exploratory study with 13 song-signers, we examined how ELMI facilitates their workflows and how song-signers leverage and receive an LLM-driven chat for translation. Participants successfully adopted ELMI to song-signing, with active discussions throughout. They also reported improved confidence and independence in their translations, finding ELMI encouraging, constructive, and informative. We discuss research and design implications for accessible and culturally sensitive song-signing translation tools.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext df8ea86c-80f6-43d6-87da-a615345fad57Cited by top-tier papers1
Ask how each one uses itBuilds on6
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language TranslationRyan Wong, Necati Cihan Camgöz, Richard BowdenICLR 2024 · 58 citations
- "In this online environment, we're limited": : Exploring Inclusive Video Conferencing Design for SignersJazz Rui Xia Ang, Ping Liu, Emma McDonnell, Sarah CoppolaCHI 2022 · 41 citations
- Understanding tensions in music accessibility through song signing for and with d/Deaf and Non-d/Deaf personsSuhyeon Yoo, Georgianna Lin, Hyeon Jeong Byeon, Amy S. Hwang et al.CHI 2023 · 17 citations
- Sign Language Transformers: Joint End-to-End Sign Language Recognition and TranslationNecati Cihan Camgöz, Oscar Koller, Simon Hadfield, Richard BowdenCVPR 2020
Related papers
- Towards AI-driven Sign Language Generation with Non-manual MarkersHan Zhang, Rotem Shalev-Arkushin, Vasileios Baltatzis, Connor Gillis et al.CHI 2025 · 12 citations
- Exploring the Potential of Music Generative AI for Music-Making by Deaf and Hard of Hearing PeopleYoujin Choi, JaeYoung Moon, Jinyoung Yoo, Jin-Hyuk HongCHI 2025 · 15 citations
- Social App Accessibility for Deaf SignersKelly Mack, Danielle Bragg, Meredith Ringel Morris, Maarten W. Bos et al.CSCW 2020 · 52 citations
- SCOPE: Sign Language Contextual Processing with Embedding from LLMsYuqi Liu, Wenqian Zhang, Sihan Ren, Chengyu Huang et al.AAAI 2025 · 7 citations
- MAVL: A Multilingual Audio-Video Lyrics Dataset for Animated Song TranslationWoohyun Cho, Youngmin Kim, Sunghyun Lee, Youngjae YuEMNLP 2025
