ACL Findings2021
VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding
Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer
Abstract
We present a simplified, task-agnostic multimodal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single crossmodal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
42 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning cites VideoBERT: A Joint Model for Video and Language Representation Learning
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning cites VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- Unified Vision-Language Pre-Training for Image Captioning and VQA cites VideoBERT: A Joint Model for Video and Language Representation Learning
- VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset cites Flamingo: a Visual Language Model for Few-Shot Learning
- EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone cites VideoBERT: A Joint Model for Video and Language Representation Learning
- EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone cites VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations cites VideoBERT: A Joint Model for Video and Language Representation Learning
- Flamingo: a Visual Language Model for Few-Shot Learning cites Unified Vision-Language Pre-Training for Image Captioning and VQA
- Flamingo: a Visual Language Model for Few-Shot Learning cites VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- Flamingo: a Visual Language Model for Few-Shot Learning cites VideoBERT: A Joint Model for Video and Language Representation Learning
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding cites VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding cites VideoBERT: A Joint Model for Video and Language Representation Learning
Cited by top-tier papers42
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text UnderstandingHu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko et al.EMNLP 2021 · 399 citations
- ST-Adapter: Parameter-Efficient Image-to-Video Transfer LearningJunting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao et al.NeurIPS 2022 · 290 citations
- VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetSihan Chen, Handong Li, Qunbo Wang, Zijia Zhao et al.NeurIPS 2023 · 246 citations
- EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the BackboneShraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin et al.ICCV 2023 · 152 citations
Builds on12
- VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video ClipsAntoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi et al.ICCV 2019 · 1,437 citations
- VideoBERT: A Joint Model for Video and Language Representation LearningChen Sun, Austin Myers, Carl Vondrick, Kevin Murphy et al.ICCV 2019 · 1,396 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- Unified Vision-Language Pre-Training for Image Captioning and VQALuowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu et al.AAAI 2020 · 1,047 citations
Related papers
- OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning FrameworkPeng Wang, An Yang, Rui Men, Junyang Lin et al.ICML 2022 · 1,058 citations
- mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and VideoHaiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi et al.ICML 2023 · 237 citations
- End-to-end Generative Pretraining for Multimodal Video CaptioningPaul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, Cordelia SchmidCVPR 2022 · 152 citations
- TeachText: CrossModal Generalized Distillation for Text-Video RetrievalIoana Croitoru, Simion-Vlad Bogolin, Marius Leordeanu, Hailin Jin et al.ICCV 2021 · 147 citations
- Stitching Segments and Sentences towards Generalization in Video-Text Pre-trainingFan Ma, Xiaojie Jin, Heng Wang, Jingjia Huang et al.AAAI 2024 · 8 citations
