Lune

EMNLP Findings2023

ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition

Xiujiao Li, Guanglu Sun, Xinyu Liu

Abstract

Multimodal Named Entity Recognition (MNER) uses visual information to improve the performance of text-only Named Entity Recognition (NER). However, existing methods for acquiring local visual information suffer from certain limitations: (1) using an attention-based method to extract visual regions related to the text from visual regions obtained through convolutional architectures (e.g., ResNet), attention is distracted by the entire image, rather than being fully focused on the visual regions most relevant to the text; (2) using an object detection-based (e.g., Mask R-CNN) method to detect visual object regions related to the text, object detection has a limited range of recognition categories. Moreover, the visual regions obtained by object detection may not correspond to the entities in the text. In summary, the goal of these methods is not to extract the most relevant visual regions for the entities in the text. The visual regions obtained by these methods may be redundant or insufficient for the entities in the text. In this paper, we propose an Entity Spans Position Visual Regions (ESPVR) module to obtain the most relevant visual regions corresponding to the entities in the text. Experiments show that our proposed approach can achieve the SOTA on Twitter-2017 and competitive results on Twitter-2015.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance cites Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer
  • RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER cites Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on5

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  2. Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal TransformerJianfei Yu, Jing Jiang, Li Yang, Rui XiaACL 2020 · 260 citations
  3. Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual GuidanceDong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu et al.AAAI 2021 · 240 citations
  4. RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NERLin Sun, Jiquan Wang, Kai Zhang, Yindu Su et al.AAAI 2021 · 189 citations
  5. Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media PostsZhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen et al.ACM MM 2020 · 139 citations

Related papers

  1. Query Prior Matters: A MRC Framework for Multimodal Named Entity RecognitionMeihuizi Jia, Xin Shen, Lei Shen, Jinhui Pang et al.ACM MM 2022 · 45 citations
  2. MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query GroundingMeihuizi Jia, Lei Shen, Xin Shen, Lejian Liao et al.AAAI 2023 · 68 citations
  3. A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity RecognitionBaohang Zhou, Ying Zhang, Kehui Song, Wenya Guo et al.EMNLP 2022 · 15 citations
  4. Hierarchical Aligned Multimodal Learning for NER on Tweet PostsPeipei Liu, Hong Li, Yimo Ren, Jie Liu et al.AAAI 2024 · 11 citations
  5. Named Entity and Relation Extraction with Multi-Modal RetrievalXinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie et al.EMNLP Findings 2022 · 54 citations
Dusk over the sea between two cliffs drawn in fine vertical lines