EMNLP Findings2025
A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks
Haorui Yu, Ramon Ruiz-Dolz, Qiufeng Yi
Abstract
This study aims to test and evaluate the capabilities and characteristics of current mainstream Visual Language Models (VLMs) in generating critiques for traditional Chinese painting. To achieve this, we first developed a quantitative framework for Chinese painting critique. This framework was constructed by extracting multi-dimensional evaluative features covering evaluative stance, feature focus, and commentary quality from human expert critiques using a zero-shot classification model. Based on these features, several representative critic personas were defined and quantified. This framework was then employed to evaluate selected VLMs such as Llama, Qwen, or Gemini. The experimental design involved persona-guided prompting to assess the VLM's ability to generate critiques from diverse perspectives. Our findings reveal the current performance levels, strengths, and areas for improvement of VLMs in the domain of art critique, offering insights into their potential and limitations in complex semantic understanding and content generation tasks. The code used for our experiments can be publicly accessed at: https: //github.com/yha9806/VULCA-EMNLP2025.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on8
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Ferret: Refer and Ground Anything Anywhere at Any GranularityHaoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du et al.ICLR 2024 · 515 citations
- RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language ModelsNoah Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu et al.ACL Findings 2024 · 120 citations
- GalleryGPT: Analyzing Paintings with Large Multimodal ModelsYi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu et al.ACM MM 2024 · 35 citations
Related papers
- Capabilities and Evaluation Biases of Large Language Models in Classical Chinese Poetry Generation: A Case Study on Tang PoetryBolei Ma, Yina Yao, Anna-Carolina HaenschACL Findings 2026
- Can MLLMs Understand the Deep Implication Behind Chinese Images?Chenhao Zhang, Xi Feng, Yuelin Bai, Xeron Du et al.ACL 2025
- VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial UnderstandingZhaowei Liu, Xin Guo, Haotian Xia, Lingfeng Zeng et al.EMNLP 2025
- Poller: Are LLMs Suitable for Evaluating Poetry Understanding Task?Shanshan Wang, Derek F. Wong, Jingming Yao, Lidia S. ChaoACL Findings 2026
- AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language ModelsYuhang Wu, Wenmeng Yu, Yean Cheng, Yan Wang et al.ACL 2025 · 9 citations
