ACL Findings2024
Unveiling the Power of Integration: Block Diagram Summarization through Local-Global Fusion
Shreyanshu Bhushan, Eun-Soo Jung, Minho Lee
摘要
Block Diagrams play an essential role in visualizing the relationships between components or systems. Generating summaries of block diagrams is important for document understanding or question answering (QA) tasks by providing concise overviews of complex systems. However, it's a challenging task as it requires compressing complex relationships into informative descriptions. In this paper, we present "BlockNet", a fusion framework that summarizes block diagrams by integrating local and global information, catering to both English and Korean languages. Additionally, we introduce a new multilingual method to produce block diagram data, resulting in a high-quality dataset called "BD-EnKo". In BlockNet, we develop "BlockSplit", an Optical Character Recognition (OCR) based algorithm employing the divide-and-conquer principle for local information extraction. We train an OCR-free transformer architecture for global information extraction using BD-EnKo and public data. To assess the effectiveness of our model, we conduct thorough experiments on different datasets. The assessment shows that BlockNet surpasses all previous methods and models, including GPT-4V, for block diagram summarization. 1
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 2 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision 引用了 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows 引用了 An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
引用它的顶会 Paper2
它引用的 Paper11
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu 等ICCV 2021 · 31,683 次引用
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida 等NeurIPS 2022 · 24,707 次引用
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn 等ICLR 2021 · 21,477 次引用
- ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 次引用
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad 等ACL 2020 · 1,224 次引用
相关 Paper
- BLOCSUM: Block Scope-based Source Code Summarization via Shared Block RepresentationYunSeok Choi, Hyojun Kim, Jee-Hyong LeeACL Findings 2023 · 2 次引用
- Assist Non-native Viewers: Multimodal Cross-Lingual Summarization for How2 VideosNayu Liu, Kaiwen Wei, Xian Sun, Hongfeng Yu 等EMNLP 2022 · 10 次引用
- BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation PretrainingMinJun Kim, Seungwoo Song, Youhan Lee, Haneol Jang 等AAAI 2024 · 10 次引用
- ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart UnderstandingJovana Kondic, Pengyuan Li, Dhiraj Joshi, Isaac Sanchez 等CVPR 2026 · 7 次引用
- VTLayout: A Multi-Modal Approach for Video Text LayoutYuxuan Zhao, Jin Ma, Zhongang Qi, Zehua Xie 等ACM MM 2023 · 1 次引用
