Long Short View Feature Decomposition via Contrastive Video Representation Learning
Abstract
Self-supervised video representation methods typically focus on the representation of temporal attributes in videos. However, the role of stationary versus non-stationary attributes is less explored: Stationary features, which remain similar throughout the video, enable the prediction of video-level action classes. Non-stationary features, which represent temporally varying attributes, are more beneficial for downstream tasks involving more fine-grained temporal understanding, such as action segmentation. We argue that a single representation to capture both types of features is sub-optimal, and propose to decompose the representation space into stationary and non-stationary features via contrastive learning from long and short views, i.e. long video sequences and their shorter sub-sequences. Stationary features are shared between the short and long views, while non-stationary features aggregate the short views to match the corresponding long view. To empirically verify our approach, we demonstrate that our stationary features work particularly well on an action recognition downstream task, while our non-stationary features perform better on action segmentation. Furthermore, we analyse the learned representations and find that stationary features capture more temporally stable, static attributes, while non-stationary features encompass more temporally varying ones.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Self-supervised Co-Training for Video Representation Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Self-supervised Co-Training for Video Representation Learning cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization cites Self-supervised Co-Training for Video Representation Learning
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization cites A Simple Framework for Contrastive Learning of Visual Representations
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization cites Self-supervised Video Transformer
- Self-supervised Video Transformer cites Self-supervised Co-Training for Video Representation Learning
- Self-supervised Video Transformer cites A Simple Framework for Contrastive Learning of Visual Representations
- Self-supervised Video Transformer cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Language-based Action Concept Spaces Improve Video Self-Supervised Learning cites Self-supervised Co-Training for Video Representation Learning
- Language-based Action Concept Spaces Improve Video Self-Supervised Learning cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Language-based Action Concept Spaces Improve Video Self-Supervised Learning cites Self-supervised Video Transformer
Cited by top-tier papers11
- Self-supervised Video TransformerKanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan et al.CVPR 2022 · 111 citations
- Verbs in Action: Improving verb understanding in video-language modelsLiliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman et al.ICCV 2023 · 93 citations
- Causal Temporal Representation Learning with Nonstationary Sparse TransitionXiangchen Song, Zijian Li, Guangyi Chen, Yujia Zheng et al.NeurIPS 2024 · 18 citations
- Language-based Action Concept Spaces Improve Video Self-Supervised LearningKanchana Ranasinghe, Michael S. RyooNeurIPS 2023 · 16 citations
- Tubelet-Contrastive Self-Supervision for Video-Efficient GeneralizationFida Mohammad Thoker, Hazel Doughty, Cees G. M. SnoekICCV 2023 · 13 citations
Builds on11
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Rethinking ImageNet Pre-TrainingKaiming He, Ross B. Girshick, Piotr DollárICCV 2019 · 1,188 citations
- Self-supervised Co-Training for Video Representation LearningTengda Han, Weidi Xie, Andrew ZissermanNeurIPS 2020 · 405 citations
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset BiasesSenthil Purushwalkam, Abhinav GuptaNeurIPS 2020 · 240 citations
Related papers
- No More Shortcuts: Realizing the Potential of Temporal Self-SupervisionIshan Rajendrakumar Dave, Simon Jenni, Mubarak ShahAAAI 2024 · 14 citations
- Composable Augmentation Encoding for Video Representation LearningChen Sun, Arsha Nagrani, Yonglong Tian, Cordelia SchmidICCV 2021 · 20 citations
- Contrast and Order Representations for Video Self-supervised LearningKai Hu, Jie Shao, Yuan Liu, Bhiksha Raj et al.ICCV 2021 · 76 citations
- Video Representation Learning with Graph Contrastive AugmentationJingran Zhang, Xing Xu, Fumin Shen, Yazhou Yao et al.ACM MM 2021 · 6 citations
- MaMiCo: Macro-to-Micro Semantic Correspondence for Self-supervised Video Representation LearningBo Fang, Wenhao Wu, Chang Liu, Yu Zhou et al.ACM MM 2022 · 6 citations
