Lune

EMNLP Findings2024

Towards One-to-Many Visual Question Answering

Huishan Ji, Qingyi Si, Zheng Lin, Yanan Cao, Weiping Wang

摘要

Most existing Visual Question Answering (VQA) systems are constrained to support domain-specific questions, i.e., to train different models separately for different VQA tasks, thus generalizing poorly to others. For example, models trained on the reasoning-focused dataset GQA struggle to effectively handle samples from the knowledge-emphasizing dataset OKVQA. Meanwhile, in real-world scenarios, it is user-unfriendly to restrict the domain of questions. Therefore, this paper proposes a necessary task: One-to-Many Visual Question Answering, of which the ultimate goal is to enable a single model to answer as many different domains of questions as possible by the effective integration of available VQA resources. To this end, we first investigate into ten common VQA datasets, and break the task of VQA down into the integration of three key abilities. Then, considering assorted questions rely on different VQA abilities, this paper proposes a novel dynamic Mixture of LoRAs (MoL) strategy. MoL mixes three individually trained LoRA adapters (corresponding to each VQA ability) dynamically for different samples demanding various VQA abilities. The proposed MoL strategy is verified to be highly effective by experiments, establishing SOTAs on four datasets. In addition, MoL generalizes well to three extra zeroshot datasets. Data and codes will be released.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 LoRA: Low-Rank Adaptation of Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
  • X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQA 引用了 Visual Instruction Tuning
  • Visual Instruction Tuning 引用了 Flamingo: a Visual Language Model for Few-Shot Learning
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper1

  1. X-FLoRA: Cross-modal Federated Learning with Modality-expert LoRA for Medical VQAMin Hyuk Kim, Changheon Kim, Seok Bong YooEMNLP 2025 · 2 次引用

它引用的 Paper19

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn 等ICLR 2021 · 21,477 次引用
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 18,833 次引用
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
  4. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech 等NeurIPS 2022 · 6,707 次引用
  5. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong 等NeurIPS 2023 · 4,013 次引用

相关 Paper

  1. MoLoRA: Composable Specialization via Per-Token Adapter RoutingShrey Shah, Justin WagleICML 2026 · 3 次引用
  2. MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual EncodersJiajun Cao, Yuan Zhang, Tao Huang, Ming Lu 等CVPR 2025
  3. Breaking the Barrier Between Pre-training and Fine-tuning: A Hybrid Prompting Model for Knowledge-Based VQAZhongfan Sun, Yongli Hu, Qingqing Gao, Huajie Jiang 等ACM MM 2023 · 6 次引用
  4. Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-task LearningZiyu Zhao, Yixiao Zhou, Xin Yu, Zhi Zhang 等KDD 2026 · 13 次引用
  5. Multimodal Instruction Tuning with Conditional Mixture of LoRAYing Shen, Zhiyang Xu, Qifan Wang, Yu Cheng 等ACL 2024
黄昏的海面,两侧是细线勾勒的悬崖