Lune

EMNLP Findings2023

Visual Storytelling with Question-Answer Plans

Danyang Liu, Mirella Lapata, Frank Keller

摘要

Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 4 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Language Is Not All You Need: Aligning Perception with Language Models 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
  • Language Is Not All You Need: Aligning Perception with Language Models 引用了 Multimodal Few-Shot Learning with Frozen Language Models
  • MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers 引用了 Language Models are Few-Shot Learners
  • Flamingo: a Visual Language Model for Few-Shot Learning 引用了 Language Models are Few-Shot Learners
  • Flamingo: a Visual Language Model for Few-Shot Learning 引用了 Multimodal Few-Shot Learning with Frozen Language Models
  • Multimodal Few-Shot Learning with Frozen Language Models 引用了 Language Models are Few-Shot Learners
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper4

  1. Learning to Plan and Generate Text with CitationsConstanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao 等ACL 2024 · 7 次引用
  2. TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based ReasoningYilun Zhao, Lyuhao Chen, Arman Cohan, Chen ZhaoACL 2024 · 5 次引用
  3. Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative LearningYi Bin, Junrong Liao, Yujuan Ding, Haoxuan Li 等ACM MM 2024 · 3 次引用
  4. What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific PresentationsDongqi Liu, Chenxi Whitehouse, Xi Yu, Louis Mahon 等ACL 2025

它引用的 Paper17

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah 等NeurIPS 2020 · 64,255 次引用
  2. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech 等NeurIPS 2022 · 6,707 次引用
  3. Multimodal Few-Shot Learning with Frozen Language ModelsMaria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami 等NeurIPS 2021 · 1,020 次引用
  4. Language Is Not All You Need: Aligning Perception with Language ModelsShaohan Huang, Li Dong, Wenhui Wang, Yaru Hao 等NeurIPS 2023 · 810 次引用
  5. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun 等NeurIPS 2021 · 606 次引用

相关 Paper

  1. Plot and Rework: Modeling Storylines for Visual StorytellingChi-Yang Hsu, Yun-Wei Chu, Ting-Hao 'Kenneth' Huang, Lun-Wei KuACL Findings 2021 · 20 次引用
  2. Text-Only Training for Visual StorytellingYuechen Wang, Wengang Zhou, Zhenbo Lu, Houqiang LiACM MM 2023 · 4 次引用
  3. Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational ReasoningChunpu Xu, Min Yang, Chengming Li, Ying Shen 等AAAI 2021 · 39 次引用
  4. VISIAR: Empower MLLM for Visual Story IdeationZhaoyang Xia, Somdeb Sarkhel, Md. Mehrab Tanjim, Stefano Petrangeli 等ACL Findings 2025
  5. LogiStory: A Logic-Aware Framework for Multi-Image Story VisualizationChutian Meng, Fan Ma, Chi Zhang, Jiaxu Miao 等ICLR 2026 · 3 次引用
黄昏的海面,两侧是细线勾勒的悬崖