Lune

ACL Findings2025

KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding

Ahmed Heakl, Muhammad Abdullah Sohail, Mukul Ranjan, Rania Elbadry, Ghazi Shazan Ahmad, Mohamed El-Geish, Omar Maher, Zhiqiang Shen, Fahad Shahbaz Khan, Salman H. Khan

摘要

With the growing adoption of Retrieval-Augmented Generation (RAG) in document processing, robust text recognition has become increasingly critical for knowledge extraction. While OCR (Optical Character Recognition) for English and other languages benefits from large datasets and well-established benchmarks, Arabic OCR faces unique challenges due to its cursive script, right-to-left text flow, and complex typographic and calligraphic features. We present KITAB-Bench, a comprehensive Arabic OCR benchmark that fills the gaps in current evaluation systems. Our benchmark comprises 8,809 samples across 9 major domains and 36 sub-domains, encompassing diverse document types including handwritten text, structured tables, and specialized coverage of 21 chart types for business intelligence. Our findings show that modern vision-language models (such as GPT-4, Gemini, and Qwen) outperform traditional OCR approaches (like Easy-OCR, PaddleOCR, and Surya) by an average of 60% in Character Error Rate (CER). Furthermore, we highlight significant limitations of current Arabic OCR models, particularly in PDF-to-Markdown conversion, where the best model Gemini-2.0-Flash achieves only 65% accuracy. This underscores the challenges in accurately recognizing Arabic text, including issues with complex fonts, numeral recognition errors, word elongation, and table structure detection. This work establishes a rigorous evaluation framework that can drive improvements in Arabic document analysis methods and bridge the performance gap with English OCR technologies.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 3 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper3

  1. POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question AnsweringYichen Xu, Liangyu Chen, Liang Zhang, Zihao Yue 等ACL 2026 · 2 次引用
  2. SAHM: A Benchmark for Arabic Financial and Shari'ah-Compliant ReasoningRania Elbadry, Sarfraz Ahmad, Ahmed Heakl, Dani Bouch 等ACL 2026 · 1 次引用
  3. TexOCR: Advancing Document OCR Models for Compilable Page-to-LaTeX ReconstructionChengye Wang, Lin Fu, Zexi Kuang, Yilun ZhaoACL 2026

它引用的 Paper5

  1. DETRs Beat YOLOs on Real-time Object DetectionYian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei 等CVPR 2024 · 3,046 次引用
  2. Peacock: A Family of Arabic Multimodal Large Language Models and BenchmarksFakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed 等ACL 2024 · 5 次引用
  3. TableFormer: Table Structure Understanding with TransformersAhmed S. Nassar, Nikolaos Livathinos, Maksym Lysak, Peter W. J. StaarCVPR 2022 · 5 次引用
  4. M6Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout AnalysisHiuyi Cheng, Peirong Zhang, Sihang Wu, Jiaxin Zhang 等CVPR 2023
  5. Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language ModelsMatt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi 等CVPR 2025

相关 Paper

  1. Gazelle: An Instruction Dataset for Arabic Writing AssistanceSamar Mohamed Magdy, Fakhraddin Alwajih, Sang Yun Kwon, Reem Abdel-Salam 等EMNLP Findings 2024 · 4 次引用
  2. OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented GenerationJunyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang 等ICCV 2025 · 10 次引用
  3. OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive AnnotationsLinke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu 等CVPR 2025
  4. MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal ModelsWenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz 等ICLR 2025 · 1 次引用
  5. MedAraBench: Large-scale Arabic Medical Question Answering Dataset and BenchmarkMouath Abu Daoud, Leen Kharouf, Omar El Hajj, Dana El Samad 等ICLR 2026 · 4 次引用
黄昏的海面,两侧是细线勾勒的悬崖