EMNLP Findings2023
Sound of Story: Multi-modal Storytelling with Audio
Jaeyeon Bae, Seokhoon Jeong, Seokun Kang, Namgi Han, Jae-Yon Lee, Hyounghun Kim, Taehwan Kim
Abstract
Storytelling is multi-modal in the real world. When one tells a story, one may use all of the visualizations and sounds along with the story itself. However, prior studies on storytelling datasets and tasks have paid little attention to sound even though sound also conveys meaningful semantics of the story. Therefore, we propose to extend story understanding and telling areas by establishing a new component called background sound which is story context-based audio without any linguistic information. For this purpose, we introduce a new dataset, called Sound of Story (SoS), which has paired image and text sequences with corresponding sound or background music for a story. To the best of our knowledge, this is the largest well-curated dataset for storytelling with sound. Our SoS dataset consists of 27,354 stories with 19.6 images per story and 984 hours of speech-decoupled audio such as background music and other sounds. As benchmark tasks for storytelling with sound and the dataset, we propose retrieval tasks between modalities, and audio generation tasks from image-text sequences, introducing strong baselines for them. We believe the proposed dataset and tasks may shed light on the multi-modal understanding of storytelling in terms of sound. Downloading the dataset and baseline codes for each task will be released in the link: https: //github.com/Sosdatasets/SoS_Dataset.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Adding Conditional Control to Text-to-Image Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
- Adding Conditional Control to Text-to-Image Diffusion Models cites Learning Transferable Visual Models From Natural Language Supervision
- Simple and Controllable Music Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
- Simple and Controllable Music Generation cites Learning Transferable Visual Models From Natural Language Supervision
- X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
Builds on15
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations
- Simple and Controllable Music GenerationJade Copet, Felix Kreuk, Itai Gat, Tal Remez et al.NeurIPS 2023 · 843 citations
- X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text RetrievalYiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan et al.ACM MM 2022 · 314 citations
Related papers
- AESOP: Abstract Encoding of Stories, Objects, and PicturesHareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt et al.ICCV 2021 · 19 citations
- PSM: Learning Probabilistic Embeddings for Multi-scale Zero-Shot Soundscape MappingSubash Khanal, Eric Xing, Srikumar Sastry, Aayush Dhakal et al.ACM MM 2024 · 3 citations
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion ModelsChang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang et al.CVPR 2024 · 32 citations
- MACS: Multi-source Audio-to-image Generation with Contextual Significance and Semantic AlignmentHao Zhou, Xiaobao Guo, Yuzhe Zhu, Adams Wai-Kin KongAAAI 2026 · 2 citations
- Semantic Alignment for Multimodal Large Language ModelsTao Wu, Mengze Li, Jingyuan Chen, Wei Ji et al.ACM MM 2024 · 13 citations
