ACL Findings2024
Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking
Zefeng Zhang, Jiawei Sheng, Chuang Zhang, Liangyunzhi Liangyunzhi, Wenyuan Zhang, Siqi Wang, Tingwen Liu
Abstract
Multimodal Entity Linking (MEL) aims to link ambiguous mentions in multimodal contexts to entities in a multimodal knowledge graph. A pivotal challenge is to fully leverage multielement correlations between mentions and entities to bridge modality gap and enable finegrained semantic matching. Existing methods attempt several local correlative mechanisms, relying heavily on the automatically learned attention weights, which may over-concentrate on partial correlations. To mitigate this issue, we formulate the correlation assignment problem as an optimal transport (OT) problem, and propose a novel MEL framework, namely OT-MEL, with OT-guided correlation assignment. Thereby, we exploit the correlation between multimodal features to enhance multimodal fusion, and the correlation between mentions and entities to enhance fine-grained matching. To accelerate model prediction, we further leverage knowledge distillation to transfer OT assignment knowledge to attention mechanism. Experimental results show that our model significantly outperforms previous state-of-the-art baselines and confirm the effectiveness of the OT-guided correlation assignment. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Multi-level Mixture of Experts for Multimodal Entity Linking cites Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
- Multi-level Mixture of Experts for Multimodal Entity Linking cites ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Multi-level Mixture of Experts for Multimodal Entity Linking cites Learning Transferable Visual Models From Natural Language Supervision
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
- Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Learning Transferable Visual Models From Natural Language Supervision
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity Linking cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers6
- SOTOPIA-: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social AgentsWenyuan Zhang, Tianyun Liu, Mengxiao Song, Xiaodong Li et al.ACL 2025 · 17 citations
- I2CR: Intra- and Inter-modal Collaborative Reflections for Multimodal Entity LinkingZiyan Liu, Junwen Li, Kaiwen Li, Tong Ruan et al.ACM MM 2025 · 2 citations
- Multi-level Mixture of Experts for Multimodal Entity LinkingZhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li et al.KDD 2025 · 1 citation
- SCOUT: Selective Coupling via Optimal Unbalanced Transport for Interpretable Text ClassificationJunhao Jia, Hanwen Zheng, Yueyi Wu, Huangwei Chen et al.ACL 2026 · 1 citation
- Calibrating Pseudo-Labeling with Class Distribution for Semi-supervised Text ClassificationWeiyi Yang, Richong Zhang, Junfan Chen, Jiawei ShengEMNLP 2025
Builds on20
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Align before Fuse: Vision and Language Representation Learning with Momentum DistillationJunnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty et al.NeurIPS 2021 · 2,985 citations
- ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
Related papers
- A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity LinkingShezheng Song, Shan Zhao, Chengyu Wang, Tianwei Yan et al.AAAI 2024
- Multimodal Entity Linking: A New Dataset and A BaselineJingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang et al.ACM MM 2021 · 41 citations
- OpenMEL: Unsupervised Multimodal Entity Linking Using Noise-Free Expanded Queries and Global CoherenceXinyi Zhu, Yongqi Zhang, Lei ChenVLDB 2025 · 2 citations
- DRIN: Dynamic Relation Interactive Network for Multimodal Entity LinkingShangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li et al.ACM MM 2023 · 21 citations
- MECI: Multi-Element Collaborative Interaction for Multimodal Entity LinkingJie Peng, Yongxue Shan, Yongfu Zha, Xiaodong WangSIGIR 2026
