EMNLP Findings2023
GenKIE: Robust Generative Multimodal Document Key Information Extraction
摘要
Key information extraction (KIE) from scanned documents has gained increasing attention because of its applications in various domains. Although promising results have been achieved by some recent KIE approaches, they are usually built based on discriminative models, which lack the ability to handle optical character recognition (OCR) errors and require laborious token-level labelling. In this paper, we propose a novel generative end-to-end model, named GenKIE, to address the KIE task. GenKIE is a sequence-to-sequence multimodal generative model that utilizes multimodal encoders to embed visual, layout and textual features and a decoder to generate the desired output. Well-designed prompts are leveraged to incorporate the label semantics as the weakly supervised signals and entice the generation of the key information. One notable advantage of the generative model is that it enables automatic correction of OCR errors. Besides, token-level granular annotation is not required. Extensive experiments on multiple public realworld datasets show that GenKIE effectively generalizes over different types of documents and achieves state-of-the-art results. Our experiments also validate the model's robustness against OCR errors, making GenKIE highly applicable in real-world scenarios 1 .
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition 引用了 LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
- OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition 引用了 Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework 引用了 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework 引用了 SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking 引用了 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
引用它的顶会 Paper1
它引用的 Paper10
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu 等ICCV 2021 · 31,683 次引用
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad 等ACL 2020 · 1,224 次引用
- OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin 等ICML 2022 · 1,058 次引用
- SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionZirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai 等ICLR 2022 · 950 次引用
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu 等ACM MM 2022 · 606 次引用
相关 Paper
- UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual DocumentsYifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen 等ACL 2026 · 3 次引用
- PM3-KIE: A Probabilistic Multi-Task Meta-Model for Document Key Information ExtractionBirgit Kirsch, Héctor Allende-Cid, Stefan RüpingACL Findings 2025 · 1 次引用
- Query-driven Generative Network for Document Information Extraction in the WildHaoyu Cao, Xin Li, Jiefeng Ma, Deqiang Jiang 等ACM MM 2022 · 13 次引用
- P²Net: Parallel Pointer-based Network for Key Information Extraction with Complex LayoutsKaiwen Wei, Jie Yao, Jiang Zhong, Yangyang Kang 等ACL Findings 2025 · 1 次引用
- Towards Robust Visual Information Extraction in Real World: New Dataset and Novel SolutionJiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang 等AAAI 2021 · 97 次引用
