Lune

ICLR2024Top-tier venue

COSA: Concatenated Sample Pretrained Vision-Language Foundation Model

Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, Jing Liu

Abstract

Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations. To address this issue, we propose COSA, a COncatenated SAmple pretrained vision-language foundation model. COSA jointly models visual contents and event-level temporal cues using only image-text corpora. We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effectively converts existing image-text corpora into a pseudo long-form video-paragraph corpus, enabling richer scene transformations and explicit event-description correspondence. Extensive experiments demonstrate that COSA consistently improves performance across a broad range of downstream tasks, including long-form/short-form video-text tasks and image-text tasks such as retrieval, captioning, and question answering. Notably, COSA achieves state-of-the-art results on various competitive benchmarks. Code and model are released at https://github.com/TXH-mercury/COSA.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Unhackable Temporal Reward for Scalable Video MLLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites Learning Transferable Visual Models From Natural Language Supervision
  • Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites Flamingo: a Visual Language Model for Few-Shot Learning
  • OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation cites Learning Transferable Visual Models From Natural Language Supervision
  • OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Diffusion-Inspired Truncated Sampler for Text-Video Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Diffusion-Inspired Truncated Sampler for Text-Video RetrievalJiamian Wang, Pichao Wang, Dongfang Liu, Qiang Guan et al.NeurIPS 2024 · 16 citations
  2. Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question AnsweringHaibo Wang, Chenghang Lai, Yixuan Sun, Weifeng GeACM MM 2024 · 12 citations
  3. OSKAR: Omnimodal Self-supervised Knowledge Abstraction and RepresentationMohamed Abdelfattah, Kaouther Messaoud, Alexandre AlahiNeurIPS 2025 · 1 citation
  4. Unhackable Temporal Reward for Scalable Video MLLMsEn Yu, Kangheng Lin, Liang Zhao, Yana Wei et al.ICLR 2025
  5. AdaSpark: Adaptive Sparsity for Efficient Long-Video UnderstandingHandong Li, Zikang Liu, Longteng Guo, Tongtian Yue et al.CVPR 2026

Builds on41

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  5. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations

Related papers

  1. TAP: Text-Aware Pre-Training for Text-VQA and Text-CaptionZhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin et al.CVPR 2021
  2. Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive LearningYuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu et al.NeurIPS 2022 · 91 citations
  3. OmniVL: One Foundation Model for Image-Language and Video-Language TasksJunke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo et al.NeurIPS 2022 · 205 citations
  4. mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionsChenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang et al.EMNLP 2022 · 159 citations
  5. Remodeling Semantic Relationships in Vision-Language Fine-TuningXiangyang Wu, Liu Liu, Baosheng Yu, Jiayan Qiu et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines