Lune

EMNLP Findings2021

Skim-Attention: Learning to Focus via Document Layout

Laura Nguyen, Thomas Scialom, Jacopo Staiano, Benjamin Piwowarski

Abstract

Transformer-based pre-training techniques of text and layout have proven effective in a number of document understanding tasks. Despite this success, multimodal pre-training models suffer from very high computational and memory costs. Motivated by human reading strategies, this paper presents Skim-Attention, a new attention mechanism that takes advantage of the structure of the document and its layout. Skim-Attention only attends to the 2-dimensional position of the words in a document. Our experiments show that Skim-Attention obtains a lower perplexity than prior works, while being more computationally efficient. Skim-Attention can be further combined with long-range Transformers to efficiently process long documents. We also show how Skim-Attention can be used off-the-shelf as a mask for any Pre-trained Language Model, allowing to improve their performance while restricting attention. Finally, we show the emergence of a document structure representation in Skim-Attention.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • Rethinking Attention with Performers cites Reformer: The Efficient Transformer
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Modeling Layout Reading Order as Ordering Relations for Visually-rich Document UnderstandingChong Zhang, Yi Tu, Yixi Zhao, Chenshu Yuan et al.EMNLP 2024 · 4 citations

Builds on4

  1. Reformer: The Efficient TransformerNikita Kitaev, Lukasz Kaiser, Anselm LevskayaICLR 2020 · 2,878 citations
  2. LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingYiheng Xu, Minghao Li, Lei Cui, Shaohan Huang et al.KDD 2020 · 575 citations
  3. Rethinking Attention with PerformersKrzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song et al.ICLR 2021 · 122 citations
  4. LayoutLMv2: Multi-modal Pre-training for Visually-rich Document UnderstandingYang Xu, Yiheng Xu, Tengchao Lv, Lei Cui et al.ACL 2021

Related papers

  1. SelfDoc: Self-Supervised Document Representation LearningPeizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I. Morariu et al.CVPR 2021
  2. LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document UnderstandingYi Tu, Ya Guo, Huan Chen, Jinyang TangACL 2023 · 21 citations
  3. A Simple yet Effective Layout Token in Large Language Models for Document UnderstandingZhaoqing Zhu, Chuwei Luo, Zirui Shao, Feiyu Gao et al.CVPR 2025
  4. StructuralLM: Structural Pre-training for Form UnderstandingChenliang Li, Bin Bi, Ming Yan, Wei Wang et al.ACL 2021
  5. Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM InferenceHoang Anh Duy Le, Sahil Joshi, Zeyu Yang, Zhaozhuo Xu et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines