Long Short View Feature Decomposition via Contrastive Video Representation Learning
摘要
Self-supervised video representation methods typically focus on the representation of temporal attributes in videos. However, the role of stationary versus non-stationary attributes is less explored: Stationary features, which remain similar throughout the video, enable the prediction of video-level action classes. Non-stationary features, which represent temporally varying attributes, are more beneficial for downstream tasks involving more fine-grained temporal understanding, such as action segmentation. We argue that a single representation to capture both types of features is sub-optimal, and propose to decompose the representation space into stationary and non-stationary features via contrastive learning from long and short views, i.e. long video sequences and their shorter sub-sequences. Stationary features are shared between the short and long views, while non-stationary features aggregate the short views to match the corresponding long view. To empirically verify our approach, we demonstrate that our stationary features work particularly well on an action recognition downstream task, while our non-stationary features perform better on action segmentation. Furthermore, we analyse the learned representations and find that stationary features capture more temporally stable, static attributes, while non-stationary features encompass more temporally varying ones.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 11 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Self-supervised Co-Training for Video Representation Learning 引用了 A Simple Framework for Contrastive Learning of Visual Representations
- Self-supervised Co-Training for Video Representation Learning 引用了 Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases 引用了 A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning 引用了 A Simple Framework for Contrastive Learning of Visual Representations
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization 引用了 Self-supervised Co-Training for Video Representation Learning
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization 引用了 A Simple Framework for Contrastive Learning of Visual Representations
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization 引用了 Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization 引用了 Self-supervised Video Transformer
- Self-supervised Video Transformer 引用了 Self-supervised Co-Training for Video Representation Learning
- Self-supervised Video Transformer 引用了 A Simple Framework for Contrastive Learning of Visual Representations
- Self-supervised Video Transformer 引用了 Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Language-based Action Concept Spaces Improve Video Self-Supervised Learning 引用了 Self-supervised Co-Training for Video Representation Learning
- Language-based Action Concept Spaces Improve Video Self-Supervised Learning 引用了 Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Language-based Action Concept Spaces Improve Video Self-Supervised Learning 引用了 Self-supervised Video Transformer
引用它的顶会 Paper11
- Self-supervised Video TransformerKanchana Ranasinghe, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan 等CVPR 2022 · 111 次引用
- Verbs in Action: Improving verb understanding in video-language modelsLiliane Momeni, Mathilde Caron, Arsha Nagrani, Andrew Zisserman 等ICCV 2023 · 93 次引用
- Causal Temporal Representation Learning with Nonstationary Sparse TransitionXiangchen Song, Zijian Li, Guangyi Chen, Yujia Zheng 等NeurIPS 2024 · 18 次引用
- Language-based Action Concept Spaces Improve Video Self-Supervised LearningKanchana Ranasinghe, Michael S. RyooNeurIPS 2023 · 16 次引用
- Tubelet-Contrastive Self-Supervision for Video-Efficient GeneralizationFida Mohammad Thoker, Hazel Doughty, Cees G. M. SnoekICCV 2023 · 13 次引用
它引用的 Paper11
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 次引用
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec 等NeurIPS 2020 · 9,171 次引用
- Rethinking ImageNet Pre-TrainingKaiming He, Ross B. Girshick, Piotr DollárICCV 2019 · 1,188 次引用
- Self-supervised Co-Training for Video Representation LearningTengda Han, Weidi Xie, Andrew ZissermanNeurIPS 2020 · 405 次引用
- Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset BiasesSenthil Purushwalkam, Abhinav GuptaNeurIPS 2020 · 240 次引用
相关 Paper
- No More Shortcuts: Realizing the Potential of Temporal Self-SupervisionIshan Rajendrakumar Dave, Simon Jenni, Mubarak ShahAAAI 2024 · 14 次引用
- Composable Augmentation Encoding for Video Representation LearningChen Sun, Arsha Nagrani, Yonglong Tian, Cordelia SchmidICCV 2021 · 20 次引用
- Contrast and Order Representations for Video Self-supervised LearningKai Hu, Jie Shao, Yuan Liu, Bhiksha Raj 等ICCV 2021 · 76 次引用
- Video Representation Learning with Graph Contrastive AugmentationJingran Zhang, Xing Xu, Fumin Shen, Yazhou Yao 等ACM MM 2021 · 6 次引用
- MaMiCo: Macro-to-Micro Semantic Correspondence for Self-supervised Video Representation LearningBo Fang, Wenhao Wu, Chang Liu, Yu Zhou 等ACM MM 2022 · 6 次引用
