EMNLP Findings2025
LaMP-Cap: Personalized Figure Caption Generation With Multimodal Figure Profiles
Ho Yin Sam Ng, Edward Hsu, Aashish Anantha Ramakrishnan, Branislav Kveton, Nedim Lipka, Franck Dernoncourt, Dongwon Lee, Tong Yu, Sungchul Kim, Ryan A. Rossi, Ting-Hao 'Kenneth' Huang
Abstract
Figure captions are crucial for helping readers understand and remember a figure's key message. Many models have been developed to generate these captions, helping authors compose better quality captions more easily. Yet, authors almost always need to revise generic AI-generated captions to match their writing style and the domain's style, highlighting the need for personalization. Despite language models' personalization (LaMP) advances, these technologies often focus on text-only settings and rarely address scenarios where both inputs and profiles are multimodal. This paper introduces LAMP-CAP, 1 a dataset for personalized figure caption generation with multimodal figure profiles. For each target figure, LAMP-CAP provides not only the needed inputs, such as figure images, but also up to three other figures from the same document-each with its image, caption, and figure-mentioning paragraphs-as a profile to characterize the context. Experiments with four LLMs show that using profile information consistently helps generate captions closer to the original author-written ones. Ablation studies reveal that images in the profile are more helpful than figure-mentioning paragraphs, highlighting the advantage of using multimodal profiles over text-only ones.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on14
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
- Striking a Balance: Reader Takeaways and Preferences when Integrating Text and ChartsChase Stokes, Vidya Setlur, Bridget Cogley, Arvind Satyanarayan et al.IEEE VIS 2022 · 64 citations
- SciCap: Generating Captions for Scientific FiguresTing-Yao Hsu, C. Lee Giles, Ting-Hao 'Kenneth' HuangEMNLP Findings 2021 · 59 citations
- Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line ChartsDae Hyun Kim, Vidya Setlur, Maneesh AgrawalaCHI 2021 · 52 citations
Related papers
- mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language ModelAnwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye et al.ACM MM 2024 · 15 citations
- RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language ModelsYeongtak Oh, Dohyun Chung, Juhyeon Shin, Sangha Park et al.NeurIPS 2025 · 12 citations
- CompCap: Improving Multimodal Large Language Models with Composite CaptionsXiaohui Chen, Satya Narayan Shukla, Mahmoud Azab, Aashu Singh et al.ICCV 2025 · 2 citations
- RAP: Retrieval-Augmented Personalization for Multimodal Large Language ModelsHaoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li et al.CVPR 2025
- Learning Long- and Short-Term User Literal-Preference with Multimodal Hierarchical Transformer Network for Personalized Image CaptionWei Zhang, Yue Ying, Pan Lu, Hongyuan ZhaAAAI 2020 · 31 citations
