Lune

EMNLP Findings2020

MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering

Aisha Urooj Khan, Amir Mazaheri, Niels da Vitoria Lobo, Mubarak Shah

Abstract

We present MMFT-BERT (MultiModal Fusion Transformer with BERT encodings), to solve Visual Question Answering (VQA) ensuring individual and combined processing of multiple input modalities. Our approach benefits from processing multimodal data (video and text) adopting the BERT encodings individually and using a novel transformerbased fusion method to fuse them together.

Our method decomposes the different sources of modalities, into different BERT instances with similar architectures, but variable weights. This achieves SOTA results on the TVQA dataset. Additionally, we provide TVQA-Visual, an isolated diagnostic subset of TVQA, which strictly requires the knowledge of visual (V) modality based on a human annotator's judgment. This set of questions helps us to study the model's behavior and the challenges TVQA poses to prevent the achievement of super human performance. Extensive experiments show the effectiveness and superiority of our method 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

13 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • VL-BERT: Pre-training of Generic Visual-Linguistic Representations cites Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
  • Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training cites VL-BERT: Pre-training of Generic Visual-Linguistic Representations
  • On the hidden treasure of dialog in video question answering cites TVQA+: Spatio-Temporal Grounding for Video Question Answering
  • Video Question Answering: Datasets, Algorithms and Challenges cites TVQA+: Spatio-Temporal Grounding for Video Question Answering
  • Video Question Answering: Datasets, Algorithms and Challenges cites CLEVRER: Collision Events for Video Representation and Reasoning
  • Video Question Answering: Datasets, Algorithms and Challenges cites On the hidden treasure of dialog in video question answering
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers13

  1. TubeR: Tubelet Transformer for Video Action DetectionJiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen et al.CVPR 2022 · 77 citations
  2. Video Question Answering: Datasets, Algorithms and ChallengesYaoyao Zhong, Wei Ji, Junbin Xiao, Yicong Li et al.EMNLP 2022 · 70 citations
  3. Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language ModelsJean Park, Kuk Jin Jang, Basam Alasaly, Sriharsha Mopidevi et al.AAAI 2025 · 21 citations
  4. Knowledge Mining with Scene Text for Fine-Grained RecognitionHao Wang, Junchao Liao, Tianheng Cheng, Zewen Gao et al.CVPR 2022 · 18 citations
  5. On the hidden treasure of dialog in video question answeringDeniz Engin, François Schnitzler, Ngoc Q. K. Duong, Yannis AvrithisICCV 2021 · 12 citations

Builds on4

  1. VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations
  2. Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-TrainingGen Li, Nan Duan, Yuejian Fang, Ming Gong et al.AAAI 2020 · 966 citations
  3. CLEVRER: Collision Events for Video Representation and ReasoningKexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli et al.ICLR 2020 · 584 citations
  4. TVQA+: Spatio-Temporal Grounding for Video Question AnsweringJie Lei, Licheng Yu, Tamara L. Berg, Mohit BansalACL 2020 · 173 citations

Related papers

  1. Probing Multi-modal Machine Translation with Pre-trained Language ModelYawei Kong, Kai FanACL Findings 2021 · 13 citations
  2. VD-BERT: A Unified Vision and Dialog Transformer with BERTYue Wang, Shafiq R. Joty, Michael R. Lyu, Irwin King et al.EMNLP 2020 · 68 citations
  3. Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQARonghang Hu, Amanpreet Singh, Trevor Darrell, Marcus RohrbachCVPR 2020
  4. Efficient End-to-End Video Question Answering with Pyramidal Multimodal TransformerMin Peng, Chongyang Wang, Yu Shi, Xiang-Dong ZhouAAAI 2023 · 13 citations
  5. Vx2Text: End-to-End Learning of Video-Based Text Generation From Multimodal InputsXudong Lin, Gedas Bertasius, Jue Wang, Shih-Fu Chang et al.CVPR 2021
Dusk over the sea between two cliffs drawn in fine vertical lines