Lune

ACL Findings2026

Progressive Re-ranking for Multimodal Retrieval-Augmented Generation via Curriculum Learning

Zhu Min, Yanchao Hao, Jian Liu, Shizhu He, Xi Chen

Abstract

Retrieval-augmented generation (RAG) can enhance large language models (LLMs) by providing external knowledge and helping reduce hallucinations. In multimodal RAG, however, retrieval remains challenging because a single retriever may fail to capture fine-grained multimodal semantics, and visually or semantically similar entities may still contain misleading information for answer generation. We propose a progressive multimodal re-ranking framework with curriculum learning to improve CLIPbased visual coarse-grained retrieval. Our framework progressively refines retrieval results through two stages: fine-grained sectionlevel re-ranking and multimodal section reassessment. To better align re-ranking with multimodal queries, we introduce a curriculumlearning strategy that trains the model with hard negatives that are visually or semantically similar but contain misleading information. Experiments on InfoSeek and Enc-VQA show that our method achieves state-of-the-art answer accuracy and competitive retrieval performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTOmar Khattab, Matei ZahariaSIGIR 2020 · 1,246 citations
  5. Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia EntitiesHexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal et al.ICCV 2023 · 123 citations

Related papers

  1. OMGM: Orchestrate Multiple Granularities and Modalities for Efficient Multimodal RetrievalWei Yang, Jingjing Fu, Rui Wang, Jinyu Wang et al.ACL 2025 · 11 citations
  2. ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question AnsweringAlberto Compagnoni, Marco Morini, Sara Sarto, Federico Cocchi et al.CVPR 2026 · 11 citations
  3. Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity IdentificationQian Ma, Qiong Wu, Zhengyi Zhou, Yao MaACL Findings 2026 · 1 citation
  4. SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved InformationJiashuo Sun, Jihai Zhang, Yucheng Zhou, Zhaochen Su et al.EMNLP 2024 · 2 citations
  5. CoRe-MMRAG: Cross-Source Knowledge Reconciliation for Multimodal RAGYang Tian, Fan Liu, Jingyuan Zhang, Victoria W. et al.ACL 2025 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines