EMNLP Findings2024
Not (yet) the whole story: Evaluating Visual Storytelling Requires More than Measuring Coherence, Grounding, and Repetition
Abstract
Visual storytelling consists in generating a natural language story given a temporally ordered sequence of images. This task is not only challenging for models, but also very difficult to evaluate with automatic metrics since there is no consensus about what makes a story 'good'. In this paper, we introduce a novel method that measures story quality in terms of human likeness regarding three key aspects highlighted in previous work: visual grounding, coherence, and repetitiveness. We then use this method to evaluate the stories generated by several models, showing that the foundation model LLaVA obtains the best result, but only slightly so compared to TAPM, a 50-times smaller visual storytelling model. Upgrading the visual and language components of TAPM results in a model that yields competitive performance with a relatively low number of parameters. Finally, we carry out a human evaluation study, whose results suggest that a 'good' story may require more than a human-like level of visual grounding, coherence, and repetition.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Engage for All: Making Ordinary Image Descriptions Appealing Again! cites LoRA: Low-Rank Adaptation of Large Language Models
- Engage for All: Making Ordinary Image Descriptions Appealing Again! cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Cited by top-tier papers1
Builds on15
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
Related papers
- What Makes A Good Story? Designing Composite Rewards for Visual StorytellingJunjie Hu, Yu Cheng, Zhe Gan, Jingjing Liu et al.AAAI 2020 · 73 citations
- Rethinking Radiology Report Generation: From Narrative Flow to Topic-Guided FindingsSheng Cheng, Devika SubramanianICLR 2026
- Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video GenerationYiping Wang, Xuehai He, Kuan Wang, Luyao Ma et al.CVPR 2025
- Visual Storytelling with Question-Answer PlansDanyang Liu, Mirella Lapata, Frank KellerEMNLP Findings 2023 · 1 citation
- Evaluating Visual Narrative Coherence in Story Visualization via Diversified StorylinesMinha Jhang, Kyeongman Park, Hyukhun Koh, Kyomin JungACL 2026
