Lune

EMNLP Findings2022

ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding

Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Yuhui Cao, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun

Abstract

Recent years have witnessed the rise and success of pre-training techniques in visually-rich document understanding. However, most existing methods lack the systematic mining and utilization of layout-centered knowledge, leading to sub-optimal performances. In this paper, we propose ERNIE-Layout, a novel document pre-training solution with layout knowledge enhancement in the whole workflow, to learn better representations that combine the features from text, layout, and image. Specifically, we first rearrange input sequences in the serialization stage, and then present a correlative pre-training task, reading order prediction, to learn the proper reading order of documents. To improve the layout awareness of the model, we integrate a spatial-aware disentangled attention into the multi-modal transformer and a replaced regions prediction task into the pretraining phase. Experimental results show that ERNIE-Layout achieves superior performance on various downstream tasks, setting new stateof-the-art on key information extraction, document image classification, and document question answering datasets. The code and models are publicly available at PaddleNLP 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

26 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding cites DocFormer: End-to-End Transformer for Document Understanding
  • LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding cites LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding
  • DocFormerv2: Local Features for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • DocFormerv2: Local Features for Document Understanding cites DocFormer: End-to-End Transformer for Document Understanding
  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition cites Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction
  • OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognition cites DocFormer: End-to-End Transformer for Document Understanding
  • Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction cites LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding
  • DocFormer: End-to-End Transformer for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding cites DocFormer: End-to-End Transformer for Document Understanding
  • LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding cites UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers26

  1. DocFormerv2: Local Features for Document UnderstandingSrikar Appalaraju, Peng Tang, Qi Dong, Nishant Sankaran et al.AAAI 2024 · 68 citations
  2. LayoutLLM: Layout Instruction Tuning with Large Language Models for Document UnderstandingChuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng et al.CVPR 2024 · 39 citations
  3. OMNIPARSER: A Unified Framework for Text Spotting, Key Information Extraction and Table RecognitionJianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu et al.CVPR 2024 · 29 citations
  4. LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document UnderstandingYi Tu, Ya Guo, Huan Chen, Jinyang TangACL 2023 · 21 citations
  5. Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path PredictionChong Zhang, Ya Guo, Yi Tu, Huan Chen et al.EMNLP 2023 · 19 citations

Builds on7

  1. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations
  2. K-BERT: Enabling Language Representation with Knowledge GraphWeijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang et al.AAAI 2020 · 898 citations
  3. LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingYiheng Xu, Minghao Li, Lei Cui, Shaohan Huang et al.KDD 2020 · 575 citations
  4. UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-TrainingHangbo Bao, Li Dong, Furu Wei, Wenhui Wang et al.ICML 2020 · 423 citations
  5. DocFormer: End-to-End Transformer for Document UnderstandingSrikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie et al.ICCV 2021 · 392 citations

Related papers

  1. LayoutLMv2: Multi-modal Pre-training for Visually-rich Document UnderstandingYang Xu, Yiheng Xu, Tengchao Lv, Lei Cui et al.ACL 2021
  2. UNER: A Unified Prediction Head for Named Entity Recognition in Visually-rich DocumentsYi Tu, Chong Zhang, Ya Guo, Huan Chen et al.ACM MM 2024 · 2 citations
  3. StructuralLM: Structural Pre-training for Form UnderstandingChenliang Li, Bin Bi, Ming Yan, Wei Wang et al.ACL 2021
  4. XYLayoutLM: Towards Layout-Aware Multimodal Networks For Visually-Rich Document UnderstandingZhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan et al.CVPR 2022 · 82 citations
  5. Enhancing Visually-Rich Document Understanding via Layout Structure ModelingQiwei Li, Zuchao Li, Xiantao Cai, Bo Du et al.ACM MM 2023 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines