Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flow
Chengsheng Zhang, Chenghao Sun, Zhining Xie, Xinmei Tian
Abstract
Large Vision-Language Models (LVLMs) represent a significant leap towards empathetic agents, demonstrating remarkable capabilities in emotion understanding. However, the internal mechanisms governing how LVLMs translate abstract visual stimuli into coherent emotional narratives remain largely unexplored, primarily due to the scarcity of visual counterfactuals and the diffuse nature of emotional expression. In this paper, we bridge this gap by introducing a steering-vector-based causal attribution framework tailored for descriptive emotional reasoning. To this end, we construct a specialized dataset to demystify the emotional circuits underlying the three-stage '' Adapt-Aggregate-Execute'' mechanism. Crucially, we discover a functional decoupling: visual emotional cues are aggregated in middle layers via attention heads, but are subsequently translated into narrative generation in deep layers through pathways. Guided by these insights, we regulate the emotional information routing to strengthen attention flow and amplify the semantic activation to consolidate expression. Extensive experiments on the comprehensive MER-UniBench demonstrate that our methods significantly improve performance via inference-time intervention, effectively mitigating emotional hallucinations and corroborating the causal fidelity of the discovered circuits.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Builds on25
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of ModalityWenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu et al.ACL 2020 · 376 citations
- Evaluating Object Hallucination in Large Vision-Language ModelsYifan Li, Yifan Du, Kun Zhou, Jinpeng Wang et al.EMNLP 2023 · 344 citations
Related papers
- Do LLMs “Feel”? Emotion Circuits Discovery and ControlChenxi Wang, Yixuan Zhang, Ruiji Yu, Yufei Zheng et al.ICML 2026 · 11 citations
- Locate-then-Sparsify: Attribution Guided Sparse Strategy for Visual Hallucination MitigationTiantian Dang, Chao Bi, Shufan Shen, Jinzhe Liu et al.CVPR 2026 · 2 citations
- Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and InterventionTianyun Yang, Ziniu Li, Juan Cao, Chang XuICLR 2025
- MvP-ECR: Multi-Perspective Emotion-Cause Reasoning for Empathetic DialogueYuanyuan He, Guotai Huang, Wei Li, Jiali You et al.AAAI 2026
- Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language ModelsJianghao Yin, Qin Chen, Kedi Chen, Jie Zhou et al.ICLR 2026 · 7 citations
