Lune

ACL Findings2023

Visual Coherence Loss for Coherent and Visually Grounded Story Generation

Xudong Hong, Vera Demberg, Asad B. Sayeed, Qiankun Zheng, Bernt Schiele

摘要

Local coherence is essential for long-form text generation models. We identify two important aspects of local coherence within the visual storytelling task: (1) the model needs to represent re-occurrences of characters within the image sequence in order to mention them correctly in the story; (2) character representations should enable us to find instances of the same characters and distinguish different characters. In this paper, we propose a loss function inspired by a linguistic theory of coherence for selfsupervised learning for image sequence representations. We further propose combining features from an object and a face detector to construct stronger character features. To evaluate input-output relevance that current referencebased metrics don't measure, we propose a character matching metric to check whether the models generate referring expressions correctly for characters in input image sequences. Experiments on a visual story generation dataset show that our proposed features and loss function are effective for generating more coherent and visually grounded stories.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper12

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu 等ICCV 2021 · 31,683 次引用
  2. RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language ModelsSamuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi 等EMNLP Findings 2020 · 692 次引用
  3. Content Planning for Neural Story Generation with Aristotelian RescoringSeraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph M. Weischedel, Nanyun PengEMNLP 2020 · 106 次引用
  4. MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language ModelsPeng Xu, Mostofa Patwary, Mohammad Shoeybi, Raul Puri 等EMNLP 2020 · 104 次引用
  5. PlotMachines: Outline-Conditioned Generation with Dynamic Plot State TrackingHannah Rashkin, Asli Celikyilmaz, Yejin Choi, Jianfeng GaoEMNLP 2020 · 100 次引用

相关 Paper

  1. Detecting and Grounding Important Characters in Visual StoriesDanyang Liu, Frank KellerAAAI 2023 · 11 次引用
  2. Intelligent Grimm - Open-ended Visual Storytelling via Latent Diffusion ModelsChang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang 等CVPR 2024 · 32 次引用
  3. What Makes A Good Story? Designing Composite Rewards for Visual StorytellingJunjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu 等AAAI 2020 · 73 次引用
  4. IdentityStory: Taming Your Identity-Preserving Generator for Human-Centric Story GenerationDonghao Zhou, Jingyu Lin, Guibao Shen, Quande Liu 等AAAI 2026 · 3 次引用
  5. Visual Storytelling with Question-Answer PlansDanyang Liu, Mirella Lapata, Frank KellerEMNLP Findings 2023 · 1 次引用
黄昏的海面,两侧是细线勾勒的悬崖