EMNLP Findings2023
Visual Storytelling with Question-Answer Plans
Danyang Liu, Mirella Lapata, Frank Keller
Abstract
Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Language Is Not All You Need: Aligning Perception with Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Language Is Not All You Need: Aligning Perception with Language Models cites Multimodal Few-Shot Learning with Frozen Language Models
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites Language Models are Few-Shot Learners
- Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
- Flamingo: a Visual Language Model for Few-Shot Learning cites Multimodal Few-Shot Learning with Frozen Language Models
- Multimodal Few-Shot Learning with Frozen Language Models cites Language Models are Few-Shot Learners
Cited by top-tier papers4
- Learning to Plan and Generate Text with CitationsConstanza Fierro, Reinald Kim Amplayo, Fantine Huot, Nicola De Cao et al.ACL 2024 · 7 citations
- TaPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based ReasoningYilun Zhao, Lyuhao Chen, Arman Cohan, Chen ZhaoACL 2024 · 5 citations
- Leveraging Weak Cross-Modal Guidance for Coherence Modelling via Iterative LearningYi Bin, Junrong Liao, Yujuan Ding, Haoxuan Li et al.ACM MM 2024 · 3 citations
- What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific PresentationsDongqi Liu, Chenxi Whitehouse, Xi Yu, Louis Mahon et al.ACL 2025
Builds on17
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- Multimodal Few-Shot Learning with Frozen Language ModelsMaria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami et al.NeurIPS 2021 · 1,020 citations
- Language Is Not All You Need: Aligning Perception with Language ModelsShaohan Huang, Li Dong, Wenhui Wang, Yaru Hao et al.NeurIPS 2023 · 810 citations
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun et al.NeurIPS 2021 · 606 citations
Related papers
- Plot and Rework: Modeling Storylines for Visual StorytellingChi-Yang Hsu, Yun-Wei Chu, Ting-Hao 'Kenneth' Huang, Lun-Wei KuACL Findings 2021 · 20 citations
- Text-Only Training for Visual StorytellingYuechen Wang, Wengang Zhou, Zhenbo Lu, Houqiang LiACM MM 2023 · 4 citations
- Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational ReasoningChunpu Xu, Min Yang, Chengming Li, Ying Shen et al.AAAI 2021 · 39 citations
- VISIAR: Empower MLLM for Visual Story IdeationZhaoyang Xia, Somdeb Sarkhel, Md. Mehrab Tanjim, Stefano Petrangeli et al.ACL Findings 2025
- LogiStory: A Logic-Aware Framework for Multi-Image Story VisualizationChutian Meng, Fan Ma, Chi Zhang, Jiaxu Miao et al.ICLR 2026 · 3 citations
