ACL Findings2026
Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition
Xinyu Liu, Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, Xiaojun Meng, Jiansheng Wei, Yanghua Xiao, Bo Xu
Abstract
Multimodal Named Entity Recognition relies on visual context to resolve textual ambiguities.To mitigate data scarcity, Data Augmentation (DA) has become a standard practice; however, existing methods predominantly adopt a one-size-fits-all and random perturbation paradigm, ignoring the internal state of the target model.In this paper, we first conduct a quantitative analysis, revealing that a significant portion of errors (over 30%) are model-specific, stemming from the unique biases of different architectures.To address this, we propose Memory-Guided Hard Data Augmentation, a framework designed to systematically repair these specific defects.First, we employ K-fold cross-validation to identify model-specific Hard Data.Second, we construct a Memory Tree and utilize Large Language Models (LLMs) with a clustering mechanism to induce macro-level error patterns from micro-level failures.This facilitates a paradigm shift from stateless instance-driven augmentation to a logical pattern-driven approach.Finally, we introduce an iterative augmentation mechanism that triggers recursive generation for stubborn instances that fail initial quality filters.Extensive experiments on Twitter-2015 and Twitter-2017 benchmarks demonstrate that our framework consistently yields significant performance gains across various MNER backbones.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance cites Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer
- RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER cites Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer
Builds on6
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations
- Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal TransformerJianfei Yu, Jing Jiang, Li Yang, Rui XiaACL 2020 · 260 citations
- Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual GuidanceDong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu et al.AAAI 2021 · 240 citations
- RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NERLin Sun, Jiquan Wang, Kai Zhang, Yindu Su et al.AAAI 2021 · 189 citations
Related papers
- Generative Multimodal Data Augmentation for Low-Resource Multimodal Named Entity RecognitionZiyan Li, Jianfei Yu, Jia Yang, Wenya Wang et al.ACM MM 2024 · 13 citations
- MCG-MNER: A Multi-Granularity Cross-Modality Generative Framework for Multimodal NER with InstructionJunjie Wu, Chen Gong, Ziqiang Cao, Guohong FuACM MM 2023 · 14 citations
- MELM: Data Augmentation with Masked Entity Language Modeling for Low-Resource NERRan Zhou, Xin Li, Ruidan He, Lidong Bing et al.ACL 2022 · 114 citations
- Query Prior Matters: A MRC Framework for Multimodal Named Entity RecognitionMeihuizi Jia, Xin Shen, Lei Shen, Jinhui Pang et al.ACM MM 2022 · 45 citations
- MAKAR: a Multi-Agent framework based Knowledge-Augmented Reasoning for Grounded Multimodal Named Entity RecognitionXinkui Lin, Yuhui Zhang, Yongxiu Xu, Kun Huang et al.EMNLP 2025
