Uni-DocRobust: Universal Plug-and-Play Robustness Enhancement for Multi-modal LLMs via Feature Restoration
Yuxuan Zhou, Baole Wei, Xingjian Hu, Haowei Chen, Yu Li, Xingyue Lin, Liangcai Gao, Zhi Tang
Abstract
Real-world degradations, such as noise, blur, and low resolution, significantly impair the performance of Multi-modal Large Language Models (MLLMs) in document understanding tasks. Despite recent advancements, progress in this field remains stifled by two critical bottlenecks: the scarcity of large-scale, aligned training data necessary for learning robustness, and the lack of transferable restoration solutions across diverse MLLM architectures. To bridge the data gap, we first present DocRobust-VQA, a large-scale dataset explicitly constructed to support robustness training. Comprising 189K aligned clean/corrupted document image pairs and 417K QA pairs, it provides the first substantial corpus for fine-tuning MLLMs to handle varying degradation conditions. Leveraging this data, we propose Uni-DocRobust, a universal plug-and-play framework that decouples restoration capabilities from specific visual encoders. Our method employs a frozen Universal Restoration Core pre-trained in a canonical feature space via multi-teacher distillation, which can be seamlessly integrated into target MLLMs (e.g., Qwen-VL, InternVL) through lightweight Feature Adapters. Extensive experiments demonstrate that Uni-DocRobust significantly enhances robust performance on MLLMs and enables a cost-effective ``pre-train once, deploy everywhere'' paradigm for robust MLLM deployment.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 25231739-c242-48d3-bdfc-b24e0e1c1b19Builds on26
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image MaskingYupan Huang, Tengchao Lv, Lei Cui, Yutong Lu et al.ACM MM 2022 · 606 citations
- Pix2Struct: Screenshot Parsing as Pretraining for Visual Language UnderstandingKenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu et al.ICML 2023 · 426 citations
- On Evaluating Adversarial Robustness of Large Vision-Language ModelsYunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang et al.NeurIPS 2023 · 404 citations
Related papers
- Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?Jiaqi Tang, Jianmin Chen, Youyang Zhai, Wei Wei et al.ICML 2026 · 1 citation
- Benchmarking Visual LLMs Resilience to Unanswerable Questions on Visually Rich DocumentsDavide Napolitano, Luca Cagliero, Fabrizio BattiloroAAAI 2026
- Robust-R1: Degradation-Aware Reasoning for Robust Visual UnderstandingJiaqi Tang, Jianmin Chen, Wei Wei, Xiaogang Xu et al.AAAI 2026 · 4 citations
- DocVLM: Make Your VLM an Efficient ReaderMor Shpigel Nacson, Aviad Aberdam, Roy Ganz, Elad Ben-Avraham et al.CVPR 2025
- Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language ModelsZhentao He, Can Zhang, Ziheng Wu, Zhenghao Chen et al.NeurIPS 2025 · 14 citations
