ACL Findings2026
Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images
Yichi Zhang, Zhuo Chen, Lingbing Guo, Wen Zhang, Huajun Chen
Abstract
Understanding and reasoning with abstractive information from the visual modality presents significant challenges for current multi-modal large language models (MLLMs). Among the various forms of abstractive information, Multi-Modal Relational Knowledge (MMRK), which represents abstract relational structures between multi-modal entities using node-edge formats, remains largely under-explored. In particular, STructured and Abstractive Reasoning (STAR) on such data has received little attention from the research community. To bridge the dual gaps in large-scale high-quality data and capability enhancement methodologies, this paper makes the following key contributions: (i). An automatic STAR data engine to synthesize images with MMRK to build multi-modal instructions with reliable chain-ofthought thinking for various STAR tasks and (ii). A comprehsive two-stage training framework, accompanied by Knowledge-informed GRPO (KGRPO) and a suite of evaluation protocols tailored to different STAR tasks. Based upon these contributions, we introduce STAR-64K, a dataset comprising 64K high-quality multi-modal instruction samples, and conduct experiments across 8 open-source MLLMs. Experimental results show that our two-stage enhancement framework enables smaller 3B/7B models to significantly outperform GPT-4o in STAR. Additionally, we provide in-depth analysis regarding the effectiveness of various designs, data transferability, and scalability.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Temp-R1: A Unified Autonomous Agent for Complex Temporal KGQA via Reverse Curriculum Reinforcement Learning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Cited by top-tier papers1
Builds on13
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
- SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
Related papers
- EvolvedGRPO: Unlocking Reasoning in LVLMs via Progressive Instruction EvolutionZhebei Shen, Qifan Yu, Juncheng Li, Wei Ji et al.NeurIPS 2025 · 2 citations
- Beyond Perception: Evaluating Abstract Visual Reasoning through Multi-Stage TaskYanbei Jiang, Yihao Ding, Chao Lei, Jiayang Ao et al.ACL Findings 2025 · 1 citation
- Multimodal Reasoning with Explicit Reasoning Patterns and RewardsHan Qiu, Sheng Jin, Zhongrong Zuo, Ziyue Wang, Qi She, Ling Shao, Shijian LuCVPR Findings 2026
- Monet: Reasoning in Latent Visual Space Beyond Image and LanguageQixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang et al.CVPR 2026
- Harnessing Webpage UIs for Text-Rich Visual UnderstandingJunpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu et al.ICLR 2025
