Support-Set Based Cross-Supervision for Video Grounding
Xinpeng Ding, Nannan Wang, Shiwei Zhang, De Cheng, Xiaomeng Li, Ziyuan Huang, Mingqian Tang, Xinbo Gao
Abstract
Current approaches for video grounding propose kinds of complex architectures to capture the video-text relations, and have achieved impressive improvements. However, it is hard to learn the complicated multi-modal relations by only architecture designing in fact. In this paper, we introduce a novel Support-set Based Cross-Supervision (Sscs) module which can improve existing methods during training phase without extra inference cost. The proposed Sscs module contains two main components, i.e., discriminative contrastive objective and generative caption objective. The contrastive objective aims to learn effective representations by contrastive learning, while the caption objective can train a powerful video encoder supervised by texts. Due to the co-existence of some visual entities in both ground-truth and background intervals, i.e. mutual exclusion, naively contrastive learning is unsuitable to video grounding. We address the problem by boosting the cross-supervision with the support-set concept, which collects visual information from the whole video and eliminates the mutual exclusion of entities. Combined with the original objectives, Sscs can enhance the abilities of multi-modal relation modeling for existing approaches. We extensively evaluate Sscs on three challenging datasets, and show that our method can improve current state-of-the-art methods by large margins, especially 6.35% in terms of R1@0.5 on Charades-STA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
16 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Support-set bottlenecks for video-text representation learning cites HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval cites Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
- Knowing Where to Focus: Event-aware Transformer for Video Grounding cites Support-set bottlenecks for video-text representation learning
- Knowing Where to Focus: Event-aware Transformer for Video Grounding cites Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization
- Knowing Where to Focus: Event-aware Transformer for Video Grounding cites Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
Cited by top-tier papers16
- CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say NoHualiang Wang, Yi Li, Huifeng Yao, Xiaomeng LiICCV 2023 · 171 citations
- Negative Sample Matters: A Renaissance of Metric Learning for Temporal GroundingZhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li et al.AAAI 2022 · 170 citations
- Knowing Where to Focus: Event-aware Transformer for Video GroundingJinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon et al.ICCV 2023 · 103 citations
- DetGPT: Detect What You Need via ReasoningRenjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan et al.EMNLP 2023 · 57 citations
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment RetrievalZhihang Liu, Jun Li, Hongtao Xie, Pandeng Li et al.AAAI 2024 · 49 citations
Builds on10
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video ClipsAntoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi et al.ICCV 2019 · 1,437 citations
- Learning 2D Temporal Adjacent Networks for Moment Localization with Natural LanguageSongyang Zhang, Houwen Peng, Jianlong Fu, Jiebo LuoAAAI 2020 · 579 citations
- Support-set bottlenecks for video-text representation learningMandela Patrick, Po-Yao Huang, Yuki Markus Asano, Florian Metze et al.ICLR 2021 · 269 citations
- 3C-Net: Category Count and Center Loss for Weakly-Supervised Action LocalizationSanath Narayan, Hisham Cholakkal, Fahad Shahbaz Khan, Ling ShaoICCV 2019 · 174 citations
- Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment LocalizationDaizong Liu, Xiaoye Qu, Xiao-Yang Liu, Jianfeng Dong et al.ACM MM 2020 · 115 citations
Related papers
- Reducing the Vision and Language Bias for Temporal Sentence GroundingDaizong Liu, Xiaoye Qu, Wei HuACM MM 2022 · 52 citations
- Exploiting Auxiliary Caption for Video GroundingHongxiang Li, Meng Cao, Xuxin Cheng, Yaowei Li et al.AAAI 2024 · 16 citations
- Set Prediction Guided by Semantic Concepts for Diverse Video CaptioningYifan Lu, Ziqi Zhang, Chunfeng Yuan, Peng Li et al.AAAI 2024 · 7 citations
- CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video RepresentationsMohammadreza Zolfaghari, Yi Zhu, Peter V. Gehler, Thomas BroxICCV 2021 · 160 citations
- Contrastive Learning with Expectation-Maximization for Weakly Supervised Phrase GroundingKeqin Chen, Richong Zhang, Samuel Mensah, Yongyi MaoEMNLP 2022 · 3 citations
