ACL Findings2026
E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition
Meng Zhang, Jinzhong Ning, Xiaolong Wu, Hongfei Lin, Yijia Zhang
Abstract
Grounded Multimodal Named Entity Recognition (GMNER) aims to jointly identify named entity mentions in text, predict their semantic types, and ground each entity to a corresponding visual region in an associated image. Existing approaches predominantly adopt pipelinebased architectures that decouple textual entity recognition and visual grounding, leading to error accumulation and suboptimal joint optimization. In this paper, we propose E2E-GMNER, a fully end-to-end generative framework that unifies entity recognition, semantic typing, visual grounding, and implicit knowledge reasoning within a single multimodal large language model. We formulate GM-NER as an instruction-tuned conditional generation task and incorporate chain-of-thought reasoning to enable the model to adaptively determine when visual evidence or background knowledge is informative, reducing reliance on noisy cues. To further address the instability of generative bounding box prediction, we introduce Gaussian Risk-Aware Box Perturbation (GRBP), which replaces hard box supervision with probabilistically perturbed soft targets to improve robustness against annotation noise and discretization errors. Extensive experiments on the Twitter-GMNER and Twitter-FMNERG benchmarks demonstrate that E2E-GMNER achieves highly competitive performance compared with state of the art methods, validating the effectiveness of unified endto-end optimization and noise-aware grounding supervision.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Granular Entity Mapper: Advancing Fine-grained Multimodal Named Entity Recognition and Grounding cites Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer
- Granular Entity Mapper: Advancing Fine-grained Multimodal Named Entity Recognition and Grounding cites Fine-Grained Multimodal Named Entity Recognition and Grounding with a Generative Framework
Builds on7
- Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao et al.ICLR 2024 · 1,170 citations
- Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal TransformerJianfei Yu, Jing Jiang, Li Yang, Rui XiaACL 2020 · 260 citations
- Generative Multimodal Data Augmentation for Low-Resource Multimodal Named Entity RecognitionZiyan Li, Jianfei Yu, Jia Yang, Wenya Wang et al.ACM MM 2024 · 13 citations
- Fine-Grained Multimodal Named Entity Recognition and Grounding with a Generative FrameworkJieming Wang, Ziyan Li, Jianfei Yu, Li Yang et al.ACM MM 2023 · 11 citations
- Granular Entity Mapper: Advancing Fine-grained Multimodal Named Entity Recognition and GroundingZiqi Wang, Chen Zhu, Zhi Zheng, Xinhang Li et al.EMNLP Findings 2024 · 8 citations
Related papers
- Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity RecognitionJinlong Ma, Yu Zhang, Xuefeng Bai, Kehai Chen et al.ACL Findings 2026
- MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query GroundingMeihuizi Jia, Lei Shen, Xin Shen, Lejian Liao et al.AAAI 2023 · 68 citations
- UnCo: Uncertainty-Driven Collaborative Framework of Large and Small Models for Grounded Multimodal NERJielong Tang, Yang Yang, Jianxing Yu, Zhen-Xing Wang et al.EMNLP 2025 · 3 citations
- LLMs as Bridges: Reformulating Grounded Multimodal Named Entity RecognitionJinyuan Li, Han Li, Di Sun, Jiahao Wang et al.ACL Findings 2024 · 27 citations
- MAKAR: a Multi-Agent framework based Knowledge-Augmented Reasoning for Grounded Multimodal Named Entity RecognitionXinkui Lin, Yuhui Zhang, Yongxiu Xu, Kun Huang et al.EMNLP 2025
