EMNLP Findings2024
Towards One-to-Many Visual Question Answering
摘要
Most existing Visual Question Answering (VQA) systems are constrained to support domain-specific questions, i.e., to train different models separately for different VQA tasks, thus generalizing poorly to others. For example, models trained on the reasoning-focused dataset GQA struggle to effectively handle samples from the knowledge-emphasizing dataset OKVQA. Meanwhile, in real-world scenarios, it is user-unfriendly to restrict the domain of questions. Therefore, this paper proposes a necessary task: One-to-Many Visual Question Answering, of which the ultimate goal is to enable a single model to answer as many different domains of questions as possible by the effective integration of available VQA resources. To this end, we first investigate into ten common VQA datasets, and break the task of VQA down into the integration of three key abilities. Then, considering assorted questions rely on different VQA abilities, this paper proposes a novel dynamic Mixture of LoRAs (MoL) strategy. MoL mixes three individually trained LoRA adapters (corresponding to each VQA ability) dynamically for different samples demanding various VQA abilities. The proposed MoL strategy is verified to be highly effective by experiments, establishing SOTAs on four datasets. In addition, MoL generalizes well to three extra zeroshot datasets. Data and codes will be released.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 LoRA: Low-Rank Adaptation of Large Language Models
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
- X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQA 引用了 Visual Instruction Tuning
- Visual Instruction Tuning 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
引用它的顶会 Paper1
它引用的 Paper19
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn 等ICLR 2021 · 21,477 次引用
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech 等NeurIPS 2022 · 6,707 次引用
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong 等NeurIPS 2023 · 4,013 次引用
相关 Paper
- MoLoRA: Composable Specialization via Per-Token Adapter RoutingShrey Shah, Justin WagleICML 2026 · 3 次引用
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual EncodersJiajun Cao, Yuan Zhang, Tao Huang, Ming Lu 等CVPR 2025
- Breaking the Barrier Between Pre-training and Fine-tuning: A Hybrid Prompting Model for Knowledge-Based VQAZhongfan Sun, Yongli Hu, Qingqing Gao, Huajie Jiang 等ACM MM 2023 · 6 次引用
- Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task LearningZiyu Zhao, Yixiao Zhou, Xin Yu, Zhi Zhang 等KDD 2026 · 13 次引用
- Multimodal Instruction Tuning with Conditional Mixture of LoRAYing Shen, Zhiyang Xu, Qifan Wang, Yu Cheng 等ACL 2024
