COSA: Concatenated Sample Pretrained Vision-Language Foundation Model
Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, Jing Liu
Abstract
Due to the limited scale and quality of video-text training corpus, most vision-language foundation models employ image-text datasets for pretraining and primarily focus on modeling visually semantic representations while disregarding temporal semantic representations and correlations. To address this issue, we propose COSA, a COncatenated SAmple pretrained vision-language foundation model. COSA jointly models visual contents and event-level temporal cues using only image-text corpora. We achieve this by sequentially concatenating multiple image-text pairs as inputs for pretraining. This transformation effectively converts existing image-text corpora into a pseudo long-form video-paragraph corpus, enabling richer scene transformations and explicit event-description correspondence. Extensive experiments demonstrate that COSA consistently improves performance across a broad range of downstream tasks, including long-form/short-form video-text tasks and image-text tasks such as retrieval, captioning, and question answering. Notably, COSA achieves state-of-the-art results on various competitive benchmarks. Code and model are released at https://github.com/TXH-mercury/COSA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Learning Transferable Visual Models From Natural Language Supervision
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Unhackable Temporal Reward for Scalable Video MLLMs cites Learning Transferable Visual Models From Natural Language Supervision
- Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites Learning Transferable Visual Models From Natural Language Supervision
- Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering cites Flamingo: a Visual Language Model for Few-Shot Learning
- OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation cites Learning Transferable Visual Models From Natural Language Supervision
- OSKAR: Omnimodal Self-supervised Knowledge Abstraction and Representation cites Flamingo: a Visual Language Model for Few-Shot Learning
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
- Diffusion-Inspired Truncated Sampler for Text-Video Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- Diffusion-Inspired Truncated Sampler for Text-Video RetrievalJiamian Wang, Pichao Wang, Dongfang Liu, Qiang Guan et al.NeurIPS 2024 · 16 citations
- Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question AnsweringHaibo Wang, Chenghang Lai, Yixuan Sun, Weifeng GeACM MM 2024 · 12 citations
- OSKAR: Omnimodal Self-supervised Knowledge Abstraction and RepresentationMohamed Abdelfattah, Kaouther Messaoud, Alexandre AlahiNeurIPS 2025 · 1 citation
- Unhackable Temporal Reward for Scalable Video MLLMsEn Yu, Kangheng Lin, Liang Zhao, Yana Wei et al.ICLR 2025
- AdaSpark: Adaptive Sparsity for Efficient Long-Video UnderstandingHandong Li, Zikang Liu, Longteng Guo, Tongtian Yue et al.CVPR 2026
Builds on41
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
Related papers
- TAP: Text-Aware Pre-Training for Text-VQA and Text-CaptionZhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin et al.CVPR 2021
- Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive LearningYuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu et al.NeurIPS 2022 · 91 citations
- OmniVL: One Foundation Model for Image-Language and Video-Language TasksJunke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo et al.NeurIPS 2022 · 205 citations
- mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionsChenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang et al.EMNLP 2022 · 159 citations
- Remodeling Semantic Relationships in Vision-Language Fine-TuningXiangyang Wu, Liu Liu, Baosheng Yu, Jiayan Qiu et al.AAAI 2026
