Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
Li Tao, Xueting Wang, Toshihiko Yamasaki
Abstract
We propose a self-supervised method to learn feature representations from videos. A standard approach in traditional self-supervised methods uses positive-negative data pairs to train with contrastive learning strategy. In such a case, different modalities of the same video are treated as positives and video clips from a different video are treated as negatives. Because the spatio-temporal information is important for video representation, we extend the negative samples by introducing intra-negative samples, which are transformed from the same anchor video by breaking temporal relations in video clips. With the proposed Inter-Intra Contrastive (IIC) framework, we can train spatio-temporal convolutional networks to learn video representations. There are many flexible options in our IIC framework and we conduct experiments by using several different configurations. Evaluations are conducted on video retrieval and video recognition tasks using the learned video representation. Our proposed IIC outperforms current state-of-the-art results by a large margin, such as 16.7% and 9.5% points improvements in top-1 accuracy on UCF101 and HMDB51 datasets for video retrieval, respectively. For video recognition, improvements can also be obtained on these two benchmark datasets. Code is available at https: //github.com/BestJuly/Inter-intra-video-contrastive-learning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
23 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Self-Supervised Video Representation Learning with Meta-Contrastive Network cites A Simple Framework for Contrastive Learning of Visual Representations
- Self-Supervised Video Representation Learning with Meta-Contrastive Network cites Momentum Contrast for Unsupervised Visual Representation Learning
- Self-Supervised Video Representation Learning with Meta-Contrastive Network cites Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
- Long-Short Temporal Contrastive Learning of Video Transformers cites A Simple Framework for Contrastive Learning of Visual Representations
- Long-Short Temporal Contrastive Learning of Video Transformers cites Momentum Contrast for Unsupervised Visual Representation Learning
- Hard Negative Mixing for Contrastive Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Hard Negative Mixing for Contrastive Learning cites Momentum Contrast for Unsupervised Visual Representation Learning
- A Simple Framework for Contrastive Learning of Visual Representations cites Momentum Contrast for Unsupervised Visual Representation Learning
- Momentum Contrast for Unsupervised Visual Representation Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion cites A Simple Framework for Contrastive Learning of Visual Representations
- Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion cites Momentum Contrast for Unsupervised Visual Representation Learning
- Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the Motion cites Video Cloze Procedure for Self-Supervised Spatio-Temporal Learning
- Path-sensitive code embedding via contrastive learning for software vulnerability detection cites A Simple Framework for Contrastive Learning of Visual Representations
Cited by top-tier papers23
- Hard Negative Mixing for Contrastive LearningYannis Kalantidis, Mert Bülent Sariyildiz, Noé Pion, Philippe Weinzaepfel et al.NeurIPS 2020 · 805 citations
- Path-sensitive code embedding via contrastive learning for software vulnerability detectionXiao Cheng, Guanqin Zhang, Haoyu Wang, Yulei SuiISSTA 2022 · 98 citations
- Enhancing Unsupervised Video Representation Learning by Decoupling the Scene and the MotionJinpeng Wang, Yuting Gao, Ke Li, Jianguo Hu et al.AAAI 2021 · 70 citations
- Self-Supervised Video Representation Learning with Meta-Contrastive NetworkYuanze Lin, Xun Guo, Yan LuICCV 2021 · 46 citations
- Long-Short Temporal Contrastive Learning of Video TransformersJue Wang, Gedas Bertasius, Du Tran, Lorenzo TorresaniCVPR 2022 · 44 citations
Builds on3
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Video Cloze Procedure for Self-Supervised Spatio-Temporal LearningDezhao Luo, Chang Liu, Yu Zhou, Dongbao Yang et al.AAAI 2020 · 167 citations
- Momentum Contrast for Unsupervised Visual Representation LearningKaiming He, Haoqi Fan, Yuxin Wu, Saining Xie et al.CVPR 2020
Related papers
- Time-Equivariant Contrastive Video Representation LearningSimon Jenni, Hailin JinICCV 2021 · 64 citations
- Cross-Architecture Self-supervised Video Representation LearningSheng Guo, Zihua Xiong, Yujie Zhong, Limin Wang et al.CVPR 2022 · 23 citations
- Contrastive Learning of Image Representations with Cross-Video Cycle-ConsistencyHaiping Wu, Xiaolong WangICCV 2021 · 35 citations
- Cycle-Contrast for Self-Supervised Video Representation LearningQuan Kong, Wenpeng Wei, Ziwei Deng, Tomoaki Yoshinaga et al.NeurIPS 2020 · 59 citations
- Dual Contrastive Learning for Spatio-temporal RepresentationShuangrui Ding, Rui Qian, Hongkai XiongACM MM 2022 · 20 citations
