Lune

EMNLP Findings2023

LayoutDIT: Layout-Aware End-to-End Document Image Translation with Multi-Step Conductive Decoder

Zhiyang Zhang, Yaping Zhang, Yupu Liang, Lu Xiang, Yang Zhao, Yu Zhou, Chengqing Zong

Abstract

Document image translation (DIT) aims to translate text embedded in images from one language to another. It is a challenging task that needs to understand visual layout with text semantics simultaneously. However, existing methods struggle to capture the crucial visual layout in real-world complex document images. In this work, we make the first attempt to incorporate layout knowledge into DIT in an end-to-end way. Specifically, we propose a novel Layout-aware end-to-end Document Image Translation (LayoutDIT) with multistep conductive decoder. A layout-aware encoder is first introduced to model visual layout relations with raw OCR results. Then a novel multi-step conductive decoder is unified with hidden states conduction across three stepdecoders to achieve the document translation step by step. Benefiting from the layout-aware end-to-end joint training, our LayoutDIT outperforms state-of-the-art methods with better parameter efficiency. Besides, we create a new multi-domain document image translation dataset to validate the model's generalization. Extensive experiments show that LayoutDIT has a good generalization in diverse and complex layout scenes.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document Understanding cites LayoutLM: Pre-training of Text and Layout for Document Image Understanding
  • MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation cites Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine TranslationYupu Liang, Yaping Zhang, Zhiyang Zhang, Yang Zhao et al.ACL 2025 · 6 citations
  2. MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine TranslationGengluo Li, Chengquan Zhang, Yupu Liang, Huawen Shen et al.CVPR 2026 · 6 citations
  3. Self-Modifying State Modeling for Simultaneous Machine TranslationDonglei Yu, Xiaomian Kang, Yuchen Liu, Yu Zhou et al.ACL 2024

Builds on7

  1. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  2. LayoutLM: Pre-training of Text and Layout for Document Image UnderstandingYiheng Xu, Minghao Li, Lei Cui, Shaohan Huang et al.KDD 2020 · 575 citations
  3. What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model AnalysisJeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park et al.ICCV 2019 · 551 citations
  4. LiLT: A Simple yet Effective Language-Independent Layout Transformer for Structured Document UnderstandingJiapeng Wang, Lianwen Jin, Kai DingACL 2022 · 188 citations
  5. Towards End-to-End Unified Scene Text Detection and Layout AnalysisShangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco et al.CVPR 2022 · 86 citations

Related papers

  1. A Query-Response Framework for Whole-Page Complex-Layout Document Image Translation with Relevant Regional ConcentrationZhiyang Zhang, Yaping Zhang, Yupu Liang, Zhiyuan Chen et al.ACL Findings 2025
  2. DiT: Self-supervised Pre-training for Document Image TransformerJunlong Li, Yiheng Xu, Tengchao Lv, Lei Cui et al.ACM MM 2022 · 184 citations
  3. StrucTexTv2: Masked Visual-Textual Prediction for Document Image Pre-trainingYuechen Yu, Yulin Li, Chengquan Zhang, Xiaoqiang Zhang et al.ICLR 2023 · 18 citations
  4. ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document UnderstandingQiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo et al.EMNLP Findings 2022 · 67 citations
  5. Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware TrainingGengluo Li, Pengyuan Lyu, Chengquan Zhang, Huawen Shen et al.CVPR 2026 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines