Multi-Granularity Multi-Modal Knowledge Graph Representation Learning via Subgraph-Aware Adaptive Fusion and Hierarchical Relation Modeling
Peining Li, Meiyu Liang, Wei Huang, Junping Du, Zhe Xue, Guanhua Ye, Wu Liu, Lei Shi
Abstract
Multi-modal knowledge graphs (MMKGs) enrich traditional knowledge graphs by incorporating heterogeneous modalities such as textual descriptions and visual content, offering complementary semantic cues for knowledge reasoning. However, existing approaches often overlook the structural dependencies within each modality, apply static or coarse-grained fusion strategies, and insufficiently model relational semantics. We propose a Multi-Granularity Multi-Modal Knowledge Graph Representation Learning Method via Subgraph-aware Adaptive Fusion and Hierarchical Relation Modeling (SAFER ), which implement multi-modal knowledge representation through adaptive fusion of multi-granularity information such as multi-modal semantics, knowledge structures and relations. SAFER explicitly constructs modality-specific subgraphs and employs structure-aware graph attention networks to effectively capture intra-modal structural dependencies. We propose an adaptive multi-modal fusion mechanism, which aggregates modality-specific embeddings at the semantic level by dynamically assigning entity-specific modality weights. We design a two-stage multi-granularity knowledge relation modeling strategy, which utilizes a structure-aware multi-modal adaptive pre-fusion to preserve topological information and a relation-aware graph attention network (RGAT) post-fusion to encode relational semantics. Extensive experiments on several benchmark datasets demonstrate that the proposed SAFER significantly outperforms competitive baselines on link prediction and relation reasoning tasks.
Ask about this paper
Ask your agent about it.
Lune has read the top-tier papers around this one, so every answer names the papers it rests on.
Related papers
- MMKGR: Multi-hop Multi-modal Knowledge Graph ReasoningShangfei Zheng, Weiqing Wang, Jianfeng Qu, Hongzhi Yin et al.ICDE 2023 · 40 citations
- Multiple Heads are Better than One: Mixture of Modality Knowledge Experts for Entity Representation LearningYichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu et al.ICLR 2025 · 1 citation
- Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph CompletionXiang Chen, Ningyu Zhang, Lei Li, Shumin Deng et al.SIGIR 2022 · 227 citations
- Multimodal Reasoning with Multimodal Knowledge GraphJunlin Lee, Yequan Wang, Jing Li, Min ZhangACL 2024 · 29 citations
- DySarl: Dynamic Structure-Aware Representation Learning for Multimodal Knowledge Graph ReasoningKangzheng Liu, Feng Zhao, Yu Yang, Guandong XuACM MM 2024 · 4 citations
