Lune

CVPR2023Top-tier venue

MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models

Dohwan Ko, Joonmyung Choi, Hyeong Kyu Choi, Kyoung-Woon On, Byungseok Roh, Hyunwoo J. Kim

Abstract

Foundation models have shown outstanding performance and generalization capabilities across domains. Since most studies on foundation models mainly focus on the pretraining phase, a naive strategy to minimize a single task-specific loss is adopted for fine-tuning. However, such fine-tuning methods do not fully leverage other losses that are potentially beneficial for the target task. Therefore, we propose MEta Loss TRansformer (MELTR), a plug-in module that automatically and non-linearly combines various loss functions to aid learning the target task via auxiliary learning. We formulate the auxiliary learning as a bi-level optimization problem and present an efficient optimization algorithm based on Approximate Implicit Differentiation (AID). For evaluation, we apply our framework to various video foundation models (UniVL, Violet and All-in-one), and show significant performance gain on all four downstream tasks: text-to-video retrieval, video question answering, video captioning, and multimodal sentiment analysis. Our qualitative analyses demonstrate that MELTR adequately 'transforms' individual loss functions and 'melts' them into an effective unified loss. Code is available at https://github.com/mlvlab/ MELTR.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

11 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Prompt Learning via Meta-Regularization cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Prompt Learning via Meta-Regularization cites Learning Transferable Visual Models From Natural Language Supervision
  • VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  • VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset cites Language Models are Few-Shot Learners
  • VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset cites Learning Transferable Visual Models From Natural Language Supervision
  • VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  • VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos cites Is Space-Time Attention All You Need for Video Understanding?
  • VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • Is Space-Time Attention All You Need for Video Understanding? cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  • Is Space-Time Attention All You Need for Video Understanding? cites Language Models are Few-Shot Learners
  • Learning Transferable Visual Models From Natural Language Supervision cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Diffusion-Inspired Truncated Sampler for Text-Video Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers11

  1. VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and DatasetSihan Chen, Handong Li, Qunbo Wang, Zijia Zhao et al.NeurIPS 2023 · 246 citations
  2. Prompt Learning via Meta-RegularizationJinyoung Park, Juyeon Ko, Hyunwoo J. KimCVPR 2024 · 17 citations
  3. Diffusion-Inspired Truncated Sampler for Text-Video RetrievalJiamian Wang, Pichao Wang, Dongfang Liu, Qiang Guan et al.NeurIPS 2024 · 16 citations
  4. Comprehensive Visual Grounding for Video DescriptionWenhui Jiang, Yibo Cheng, Linxin Liu, Yuming Fang et al.AAAI 2024 · 5 citations
  5. VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in VideosBaoyu Liang, Qile Su, Shoutai Zhu, Yuchen Liang et al.AAAI 2025 · 5 citations

Builds on28

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  4. Is Space-Time Attention All You Need for Video Understanding?Gedas Bertasius, Heng Wang, Lorenzo TorresaniICML 2021 · 2,927 citations
  5. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video ClipsAntoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi et al.ICCV 2019 · 1,437 citations

Related papers

  1. OmniVL: One Foundation Model for Image-Language and Video-Language TasksJunke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo et al.NeurIPS 2022 · 205 citations
  2. LAVENDER: Unifying Video-Language Understanding as Masked Language ModelingLinjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin et al.CVPR 2023
  3. Emu: Generative Pretraining in MultimodalityQuan Sun, Qiying Yu, Yufeng Cui, Fan Zhang et al.ICLR 2024 · 161 citations
  4. Foundation Model is Efficient Multimodal Multitask Model SelectorFanqing Meng, Wenqi Shao, Zhanglin Peng, Chonghe Jiang et al.NeurIPS 2023 · 26 citations
  5. Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language TasksWenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck et al.CVPR 2023
Dusk over the sea between two cliffs drawn in fine vertical lines