VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answering
Chun-Mei Feng, Yang Bai, Tao Luo, Zhen Li, Salman H. Khan, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu
Abstract
Albeit progress has been made in Composed Image Retrieval (CIR), we empirically find that a certain percentage of failure retrieval results are not consistent with their relative captions. To address this issue, this work provides a Visual Question Answering (VQA) perspective to boost the performance of CIR. The resulting VQA4CIR is a post-processing approach and can be directly plugged into existing CIR methods. Given the top-C retrieved images by a CIR method, VQA4CIR aims to decrease the adverse effect of the failure retrieval results being inconsistent with the relative caption. To find the retrieved images inconsistent with the relative caption, we resort to the "QA generation → VQA" self-verification pipeline. For QA generation, we suggest fine-tuning LLM (e.g., LLaMA) to generate several pairs of questions and answers from each relative caption. We then fine-tune LVLM (e.g., LLaVA) to obtain the VQA model. By feeding the retrieved image and question to the VQA model, one can find the images inconsistent with relative caption when the answer by VQA is inconsistent with the answer in the QA pair. Consequently, the CIR performance can be boosted by modifying the ranks of inconsistently retrieved images. Experimental results show that our proposed method outperforms state-of-the-art CIR methods on the CIRR and Fashion-IQ datasets.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Language Models are Few-Shot Learners
- Duplex Rewards Optimization for Test-Time Composed Image Retrieval cites Training language models to follow instructions with human feedback
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Training language models to follow instructions with human feedback
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks cites Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents
- Multi-Schema Proximity Network for Composed Image Retrieval cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Language Models are Few-Shot Learners
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers5
- MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video HaystacksSanjoy Chowdhury, Mohamed Elmoghany, Yohan Abeysinghe, Junjie Fei et al.NeurIPS 2025 · 14 citations
- VisRet: Visualization Improves Knowledge-Intensive Text-to-Image RetrievalDi Wu, Yixin Wan, Kai-Wei ChangACL 2026 · 3 citations
- Multi-Schema Proximity Network for Composed Image RetrievalJiangming Shi, Xiangbo Yin, Yeyun Chen, Yachao Zhang et al.ICCV 2025 · 1 citation
- Duplex Rewards Optimization for Test-Time Composed Image RetrievalHaoliang Zhou, Feifei Zhang, Changsheng XuAAAI 2026
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ DocumentsJun Chen, Dannong Xu, Junjie Fei, Chun-Mei Feng et al.CVPR 2025
Builds on20
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
Related papers
- FIRE-CIR: Fine-grained Reasoning for Composed Fashion Image RetrievalFrançois Gardères, Camille-Sovanneary Gauthier, Jean Ponce, Shizhe ChenCVPR Findings 2026
- ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image RetrievalTianyu Yang, ChenWei He, Xiangzhao Hao, Tianyue Wang et al.CVPR 2026 · 3 citations
- Sentence-level Prompts Benefit Composed Image RetrievalYang Bai, Xinxing Xu, Yong Liu, Salman Khan et al.ICLR 2024 · 75 citations
- Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and NegativesZhangchi Feng, Richong Zhang, Zhijie NieACM MM 2024 · 14 citations
- Leveraging Large Vision-Language Model as User Intent-Aware Encoder for Composed Image RetrievalZelong Sun, Dong Jing, Guoxing Yang, Nanyi Fei et al.AAAI 2025 · 13 citations
