Active Perceptual Inference: A Corticothalamic-Inspired Dynamic Nested Recurrent Network for Multimodal Sentiment Analysis with Incomplete Data
Yujuan Zhang, Qing Li, Ziyu Li, Xiuxing Li, Zhuo Wang, Mengrui Xu, Xia Wu
Abstract
Random frame-level data missing is a critical challenge in multimodal sentiment analysis. Existing methods are largely limited to passive completion via single-pass feedforward connections and static cross-modal fusion, which struggle to generate high-quality completed features. However, the brain is not a passive recipient of external information but a dynamic system for active perceptual inference. Its core lies in the dynamic nested recurrents formed by intra-cortical recurrent completion mechanisms and corticothalamic circuits, which iteratively perform perceptual inference. Inspired by this, we propose the Dynamic Nested Recurrent Network (DNRNet). It is the first to introduce recurrent inference into the data completion task, achieving a paradigm shift from passive completion to active perceptual inference. Its local recurrent loop simulates intra-cortical recurrent pattern completion to perform perceptual inference and generate local correction features. The global recurrent loop simulates the modulatory function of the thalamus, calculating modality confidence to dynamically weight and integrate cross-modal information, generating global correction features. The local and global correction features are fused to obtain the completion signal, which is then combined with the input features of the current iteration to serve as the input for the next iteration. Experiments on the MOSI, MOSEI, and SIMS datasets demonstrate that DNRNet achieves an average accuracy improvement of 1.5%–2.0% over baseline models across all missing rates, validating the superiority of the brain-inspired approach in complex missing data scenarios.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext aab54d5a-74cf-40a5-8480-0ea86059f28fBuilds on21
- MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment AnalysisDevamanyu Hazarika, Roger Zimmermann, Soujanya PoriaACM MM 2020 · 1,037 citations
- Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment AnalysisWenmeng Yu, Hua Xu, Ziqi Yuan, Jiele WuAAAI 2021 · 737 citations
- CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of ModalityWenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu et al.ACL 2020 · 376 citations
- Transformer-based Feature Reconstruction Network for Robust Multimodal Sentiment AnalysisZiqi Yuan, Wei Li, Hua Xu, Wenmeng YuACM MM 2021 · 186 citations
- Incomplete Multimodality-Diffused Emotion RecognitionYuanzhi Wang, Yong Li, Zhen CuiNeurIPS 2023 · 155 citations
Related papers
- HiTF: A Hippocampal-Thalamic Function-Inspired Framework for Robust Multimodal Sentiment Analysis with Incomplete DataYujuan Zhang, Qing Li, Xiuxing Li, Zhuo Wang et al.KDD 2026
- Recovering Coherent Affective Patterns: Addressing Modality Missing in Multimodal Sentiment AnalysisHuiting Huang, Tieliang Gong, Kai He, Wen Wen et al.AAAI 2026
- Sample-specific Modality Diagnosis and Cross-modal Enhancement for Incomplete Multimodal RepresentationsJunsong Chen, Jiyuan Liu, Suyuan Liu, Wei Zhang et al.AAAI 2026
- Mitigating Inconsistencies in Multimodal Sentiment Analysis under Uncertain Missing ModalitiesJiandian Zeng, Jiantao Zhou, Tianyi LiuEMNLP 2022 · 29 citations
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing ModalitiesMingcheng Li, Dingkang Yang, Yuxuan Lei, Shunli Wang et al.AAAI 2024 · 71 citations
