EMNLP Findings2022
Named Entity and Relation Extraction with Multi-Modal Retrieval
Xinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie, Kewei Tu, Wei Lu
Abstract
Multi-modal named entity recognition (NER) and relation extraction (RE) aim to leverage relevant image information to improve the performance of NER and RE.Most existing efforts largely focused on directly extracting potentially useful information from images (such as pixel-level features, identified objects, and associated captions).However, such extraction processes may not be knowledge aware, resulting in information that may not be highly relevant.In this paper, we propose a novel Multi-modal Retrieval based framework (MoRe).MoRe contains a text retrieval module and an imagebased retrieval module, which retrieve related knowledge of the input text and image in the knowledge corpus respectively.Next, the retrieval results are sent to the textual and visual models respectively for predictions.Finally, a Mixture of Experts (MoE) module combines the predictions from the two models to make the final decision.Our experiments show that both our textual model and visual model can achieve state-of-the-art performance on four multi-modal NER datasets and one multimodal RE dataset.With MoE, the model performance can be further improved and our analysis demonstrates the benefits of integrating both textual and visual cues for such tasks.1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Prototype-Guided Multimodal Relation Extraction based on Entity Attributes cites Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction
- Improving Language Models by Retrieving from Trillions of Tokens cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Improving Language Models by Retrieving from Trillions of Tokens cites Retrieval Augmented Language Model Pre-Training
Cited by top-tier papers6
- Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation ExtractionXuming Hu, Junzhe Chen, Aiwei Liu, Shiao Meng et al.ACM MM 2023 · 30 citations
- MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark EvaluationLiang He, Hongke Wang, Yongchang Cao, Zhen Wu et al.ACM MM 2023 · 17 citations
- Shapley Value-based Contrastive Alignment for Multimodal Information ExtractionWen Luo, Yu Xia, Tianshu Shen, Sujian LiACM MM 2024 · 10 citations
- Aspect Enhancement and Text Simplification in Multimodal Aspect-Based Sentiment Analysis for Multi-Aspect and Multi-Sentiment ScenariosLinlin Zhu, Heli Sun, Qunshu Gao, Yuze Liu et al.AAAI 2025 · 10 citations
- Prototype-Guided Multimodal Relation Extraction based on Entity AttributesZefan Zhang, Weiqi Zhang, Yanhui Li, Tian BaiAAAI 2025 · 8 citations
Builds on17
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
- Improving Language Models by Retrieving from Trillions of TokensSebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai et al.ICML 2022 · 1,629 citations
- K-BERT: Enabling Language Representation with Knowledge GraphWeijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang et al.AAAI 2020 · 898 citations
- Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations
Related papers
- Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation ExtractionQian Li, Shu Guo, Cheng Ji, Xutan Peng et al.ACL Findings 2023 · 22 citations
- Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NERFei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing et al.ACM MM 2022 · 59 citations
- MCG-MNER: A Multi-Granularity Cross-Modality Generative Framework for Multimodal NER with InstructionJunjie Wu, Chen Gong, Ziqiang Cao, Guohong FuACM MM 2023 · 14 citations
- Query Prior Matters: A MRC Framework for Multimodal Named Entity RecognitionMeihuizi Jia, Xin Shen, Lei Shen, Jinhui Pang et al.ACM MM 2022 · 45 citations
- MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query GroundingMeihuizi Jia, Lei Shen, Xin Shen, Lejian Liao et al.AAAI 2023 · 68 citations
