Lune

EMNLP Findings2023

GenKIE: Robust Generative Multimodal Document Key Information Extraction

Panfeng Cao, Ye Wang, Qiang Zhang, Zaiqiao Meng

摘要

Key information extraction (KIE) from scanned documents has gained increasing attention because of its applications in various domains. Although promising results have been achieved by some recent KIE approaches, they are usually built based on discriminative models, which lack the ability to handle optical character recognition (OCR) errors and require laborious token-level labelling. In this paper, we propose a novel generative end-to-end model, named GenKIE, to address the KIE task. GenKIE is a sequence-to-sequence multimodal generative model that utilizes multimodal encoders to embed visual, layout and textual features and a decoder to generate the desired output. Well-designed prompts are leveraged to incorporate the label semantics as the weakly supervised signals and entice the generation of the key information. One notable advantage of the generative model is that it enables automatic correction of OCR errors. Besides, token-level granular annotation is not required. Extensive experiments on multiple public realworld datasets show that GenKIE effectively generalizes over different types of documents and achieves state-of-the-art results. Our experiments also validate the model's robustness against OCR errors, making GenKIE highly applicable in real-world scenarios 1 .

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition 引用了 LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition 引用了 Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework 引用了 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework 引用了 SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  • LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking 引用了 BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper1

  1. OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table RecognitionJianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu 等CVPR 2024 · 29 次引用

它引用的 Paper10

  1. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu 等ICCV 2021 · 31,683 次引用
  2. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad 等ACL 2020 · 1,224 次引用
  3. OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin 等ICML 2022 · 1,058 次引用
  4. SimVLM: Simple Visual Language Model Pretraining with Weak SupervisionZirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai 等ICLR 2022 · 950 次引用
  5. LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu 等ACM MM 2022 · 606 次引用

相关 Paper

  1. UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual DocumentsYifan Ji, Zhipeng Xu, Zhenghao Liu, Zulong Chen 等ACL 2026 · 3 次引用
  2. PM3-KIE: A Probabilistic Multi-Task Meta-Model for Document Key Information ExtractionBirgit Kirsch, Héctor Allende-Cid, Stefan RüpingACL Findings 2025 · 1 次引用
  3. Query-driven Generative Network for Document Information Extraction in the WildHaoyu Cao, Xin Li, Jiefeng Ma, Deqiang Jiang 等ACM MM 2022 · 13 次引用
  4. P²Net: Parallel Pointer-based Network for Key Information Extraction with Complex LayoutsKaiwen Wei, Jie Yao, Jiang Zhong, Yangyang Kang 等ACL Findings 2025 · 1 次引用
  5. Towards Robust Visual Information Extraction in Real World: New Dataset and Novel SolutionJiapeng Wang, Chongyu Liu, Lianwen Jin, Guozhi Tang 等AAAI 2021 · 97 次引用
黄昏的海面,两侧是细线勾勒的悬崖