Lune

ACL Findings2025

A Bounding Box is Worth One Token - Interleaving Layout and Text in a Large Language Model for Document Understanding

Jinghui Lu, Haiyang Yu, Yanjie Wang, Yongjie Ye, Jingqun Tang, Ziwei Yang, Binghong Wu, Qi Liu, Hao Feng, Han Wang, Hao Liu, Can Huang

Abstract

Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM) for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

12 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • A Token-Level Text Image Foundation Model for Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • A Token-Level Text Image Foundation Model for Document Understanding cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
  • A Token-Level Text Image Foundation Model for Document Understanding cites DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming
  • LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy cites Language Is Not All You Need: Aligning Perception with Language Models
  • Grounding Multimodal Large Language Models to the World cites Language Is Not All You Need: Aligning Perception with Language Models
  • DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming cites ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers12

  1. TabPedia: Towards Comprehensive Visual Table Understanding with Concept SynergyWeichao Zhao, Hao Feng, Qi Liu, Jingqun Tang et al.NeurIPS 2024 · 97 citations
  2. DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual SlimmingJiaxin Zhang, Wentao Yang, Songxuan Lai, Zecheng Xie et al.AAAI 2025 · 39 citations
  3. PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity RecognitionJinghui Lu, Yanjie Wang, Ziwei Yang, Xuejing Liu et al.NeurIPS 2024 · 22 citations
  4. Constructive Distortion: Improving MLLMs with Attention-Guided Image WarpingDwip Dalal, Gautam Vashishtha, Utkarsh Mishra, Jeonghwan Kim et al.ICLR 2026 · 17 citations
  5. A Token-Level Text Image Foundation Model for Document UnderstandingTongkun Guan, Zining Wang, Pei Fu, Zhengtao Guo et al.ICCV 2025 · 5 citations

Builds on25

  1. Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao et al.ICLR 2024 · 1,170 citations
  2. Language Is Not All You Need: Aligning Perception with Language ModelsShaohan Huang, Li Dong, Wenhui Wang, Yaru Hao et al.NeurIPS 2023 · 810 citations
  3. LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu et al.ACM MM 2022 · 606 citations
  4. LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingYiheng Xu, Minghao Li, Lei Cui, Shaohan Huang et al.KDD 2020 · 575 citations
  5. ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical ReasoningAhmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty et al.ACL Findings 2022 · 389 citations

Related papers

  1. A Simple yet Effective Layout Token in Large Language Models for Document UnderstandingZhaoqing Zhu, Chuwei Luo, Zirui Shao, Feiyu Gao et al.CVPR 2025
  2. DocLLM: A Layout-Aware Generative Language Model for Multimodal Document UnderstandingDongsheng Wang, Natraj Raman, Mathieu Sibue, Zhiqiang Ma et al.ACL 2024 · 37 citations
  3. LayoutLLM: Layout Instruction Tuning with Large Language Models for Document UnderstandingChuwei Luo, Yufan Shen, Zhaoqing Zhu, Qi Zheng et al.CVPR 2024 · 39 citations
  4. DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document UnderstandingWenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang et al.CVPR 2025
  5. DocVLM: Make Your VLM an Efficient ReaderMor Shpigel Nacson, Aviad Aberdam, Roy Ganz, Elad Ben-Avraham et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines