Diffusion-Inspired Truncated Sampler for Text-Video Retrieval
Jiamian Wang, Pichao Wang, Dongfang Liu, Qiang Guan, Sohail A. Dianat, Majid Rabbani, Raghuveer Rao, Zhiqiang Tao
Abstract
Prevalent text-to-video retrieval methods represent multimodal text-video data in a joint embedding space, aiming at bridging the relevant text-video pairs and pulling away irrelevant ones. One main challenge in state-of-the-art retrieval methods lies in the modality gap, which stems from the substantial disparities between text and video and can persist in the joint space. In this work, we leverage the potential of Diffusion models to address the text-video modality gap by progressively aligning text and video embeddings in a unified space. However, we identify two key limitations of existing Diffusion models in retrieval tasks: The L 2 loss does not fit the ranking problem inherent in text-video retrieval, and the generation quality heavily depends on the varied initial point drawn from the isotropic Gaussian, causing inaccurate retrieval. To this end, we introduce a new Diffusion-Inspired Truncated Sampler (DITS) that jointly performs progressive alignment and modality gap modeling in the joint embedding space. The key innovation of DITS is to leverage the inherent proximity of text and video embeddings, defining a truncated diffusion flow from the fixed text embedding to the video embedding, enhancing controllability compared to adopting the isotropic Gaussian. Moreover, DITS adopts the contrastive loss to jointly consider the relevant and irrelevant pairs, not only facilitating alignment but also yielding a discriminatively structured embedding. Experiments on five benchmark datasets suggest the state-of-the-art performance of DITS. We empirically find that DITS can also improve the structure of the CLIP embedding space. Code is available at https://github.com/Jiamian-Wang/DITS-text-video-retrieval * The work does not relate to author's position at Amazon.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
- Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval cites High-Resolution Image Synthesis with Latent Diffusion Models
- Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval cites Scalable Diffusion Models with Transformers
- Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
- Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval cites Denoising Diffusion Probabilistic Models
- BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance cites Learning Transferable Visual Models From Natural Language Supervision
- Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection cites Learning Transferable Visual Models From Natural Language Supervision
- Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action Detection cites Denoising Diffusion Probabilistic Models
- Scalable Diffusion Models with Transformers cites High-Resolution Image Synthesis with Latent Diffusion Models
- Scalable Diffusion Models with Transformers cites Learning Transferable Visual Models From Natural Language Supervision
- Scalable Diffusion Models with Transformers cites Denoising Diffusion Probabilistic Models
- Scalable Diffusion Models with Transformers cites Denoising Diffusion Implicit Models
- Temporal Calibrating and Distilling for Scene-Text Aware Text-Video Retrieval cites BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element Guidance
- Temporal Calibrating and Distilling for Scene-Text Aware Text-Video Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
- Temporal Calibrating and Distilling for Scene-Text Aware Text-Video Retrieval cites Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach
- Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video RetrievalJun Li, Xuhang Lou, Jinpeng Wang, Yuting Wang et al.CVPR 2026 · 3 citations
- BiMa: Towards Biases Mitigation for Text-Video Retrieval via Scene Element GuidanceHuy Le, Nhat Chung, Tung Kieu, Anh Nguyen et al.ACM MM 2025 · 2 citations
- Temporal Calibrating and Distilling for Scene-Text Aware Text-Video RetrievalZhiqian Zhao, Liang Li, Lei Shen, Xichun Sheng et al.AAAI 2026 · 1 citation
- Denoise and Align: Diffusion-Driven Foreground Knowledge Prompting for Open-Vocabulary Temporal Action DetectionSa Zhu, Wanqian Zhang, Lin Wang, Jinchao Zhang et al.SIGIR 2026
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric ApproachZechen Bai, Tianjun Xiao, Tong He, Pichao Wang et al.ICLR 2025
Builds on50
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
- Scalable Diffusion Models with TransformersWilliam Peebles, Saining XieICCV 2023 · 5,568 citations
Related papers
- DiffusionRet: Generative Text-Video Retrieval with Diffusion ModelPeng Jin, Hao Li, Zesen Cheng, Kehan Li et al.ICCV 2023 · 95 citations
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image CaptioningJeong Ryong Lee, Yejee Shin, Geonhui Son, Dosik HwangCVPR 2025
- Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video RetrievalJian Xiao, Zijie Song, Jialong Hu, Hao Cheng et al.NeurIPS 2025 · 3 citations
- Dual Alignment Unsupervised Domain Adaptation for Video-Text RetrievalXiaoshuai Hao, Wanqian Zhang, Dayan Wu, Fei Zhu et al.CVPR 2023
- HiT: Hierarchical Transformer with Momentum Contrast for Video-Text RetrievalSong Liu, Haoqi Fan, Shengsheng Qian, Yiru Chen et al.ICCV 2021 · 172 citations
