ACL Findings2023
Visual Coherence Loss for Coherent and Visually Grounded Story Generation
Xudong Hong, Vera Demberg, Asad B. Sayeed, Qiankun Zheng, Bernt Schiele
摘要
Local coherence is essential for long-form text generation models. We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters. In this paper, we propose a loss function inspired by a linguistic theory of coherence for selfsupervised learning for image sequence representations. We further propose combining features from an object and a face detector to construct stronger character features. To evaluate input-output relevance that current referencebased metrics don't measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
它引用的 Paper12
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu 等ICCV 2021 · 31,683 次引用
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsSamuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi 等EMNLP Findings 2020 · 692 次引用
- Content Planning for Neural Story Generation with Aristotelian RescoringSeraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph M. Weischedel, Nanyun PengEMNLP 2020 · 106 次引用
- MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language ModelsPeng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri 等EMNLP 2020 · 104 次引用
- PlotMachines: Outline-Conditioned Generation with Dynamic Plot State TrackingHannah Rashkin, Asli Celikyilmaz, Yejin Choi, Jianfeng GaoEMNLP 2020 · 100 次引用
相关 Paper
- Detecting and Grounding Important Characters in Visual StoriesDanyang Liu, Frank KellerAAAI 2023 · 11 次引用
- Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion ModelsChang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang 等CVPR 2024 · 32 次引用
- What Makes A Good Story? Designing Composite Rewards for Visual StorytellingJunjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu 等AAAI 2020 · 73 次引用
- IdentityStory: Taming Your Identity-Preserving Generator for Human-Centric Story GenerationDonghao Zhou, Jingyu Lin, Guibao Shen, Quande Liu 等AAAI 2026 · 3 次引用
- Visual Storytelling with Question-Answer PlansDanyang Liu, Mirella Lapata, Frank KellerEMNLP Findings 2023 · 1 次引用
