Lune

EMNLP Findings2023

GenKIE: Robust Generative Multimodal Document Key Information Extraction

Panfeng Cao, Ye Wang, Qiang Zhang, Zaiqiao Meng

Abstract

Key information extraction (KIE) from scanned documents has gained increasing attention because of its applications in various domains. Although promising results have been achieved by some recent KIE approaches, they are usually built based on discriminative models, which lack the ability to handle optical character recognition (OCR) errors and require laborious token-level labelling. In this paper, we propose a novel generative end-to-end model, named GenKIE, to address the KIE task. GenKIE is a sequence-to-sequence multimodal generative model that utilizes multimodal encoders to embed visual, layout and textual features and a decoder to generate the desired output. Well-designed prompts are leveraged to incorporate the label semantics as the weakly supervised signals and entice the generation of the key information. One notable advantage of the generative model is that it enables automatic correction of OCR errors. Besides, token-level granular annotation is not required. Extensive experiments on multiple public realworld datasets show that GenKIE effectively generalizes over different types of documents and achieves state-of-the-art results. Our experiments also validate the model's robustness against OCR errors, making GenKIE highly applicable in real-world scenarios 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition cites LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework cites SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  • LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table RecognitionJianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu et al.CVPR 2024 · 29 citations

Builds on10

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  3. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin et al.ICML 2022 · 1,058 citations
  4. SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionZirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai et al.ICLR 2022 · 950 citations
  5. LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu et al.ACM MM 2022 · 606 citations

Related papers

  1. UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual DocumentsYifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen et al.ACL 2026 · 3 citations
  2. PM3-KIE: A Probabilistic Multi-Task Meta-Model for Document Key Information ExtractionBirgit Kirsch, Héctor Allende-Cid, Stefan RüpingACL Findings 2025 · 1 citation
  3. Query-driven Generative Network for Document Information Extraction in the WildHaoyu Cao, Xin Li, Jiefeng Ma, Deqiang Jiang et al.ACM MM 2022 · 13 citations
  4. P²Net: Parallel Pointer-based Network for Key Information Extraction with Complex LayoutsKaiwen Wei, Jie Yao, Jiang Zhong, Yangyang Kang et al.ACL Findings 2025 · 1 citation
  5. Towards Robust Visual Information Extraction in Real World: New Dataset and Novel SolutionJiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang et al.AAAI 2021 · 97 citations
Dusk over the sea between two cliffs drawn in fine vertical lines