Mining Better Samples for Contrastive Learning of Temporal Correspondence
Sangryul Jeon, Dongbo Min, Seungryong Kim, Kwanghoon Sohn
Abstract
We present a novel framework for contrastive learning of pixel-level representation using only unlabeled video. Without the need of ground-truth annotation, our method is capable of collecting well-defined positive correspondences by measuring their confidences and well-defined negative ones by appropriately adjusting their hardness during training. This allows us to suppress the adverse impact of ambiguous matches and prevent a trivial solution from being yielded by too hard or too easy negative samples. To accomplish this, we incorporate three different criteria that ranges from a pixel-level matching confidence to a videolevel one into a bottom-up pipeline, and plan a curriculum that is aware of current representation power for the adaptive hardness of negative samples during training. With the proposed method, state-of-the-art performance is attained over the latest approaches on several video label propagation tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
14 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning cites Space-Time Correspondence as a Contrastive Random Walk
- Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning cites Video Object Segmentation Using Space-Time Memory Networks
- Space-Time Correspondence as a Contrastive Random Walk cites SuperGlue: Learning Feature Matching With Graph Neural Networks
- Space-Time Correspondence as a Contrastive Random Walk cites A Simple Framework for Contrastive Learning of Visual Representations
- Space-Time Correspondence as a Contrastive Random Walk cites Video Object Segmentation Using Space-Time Memory Networks
- Contrastive Learning for Space-time Correspondence via Self-cycle Consistency cites A Simple Framework for Contrastive Learning of Visual Representations
- Contrastive Learning for Space-time Correspondence via Self-cycle Consistency cites Space-Time Correspondence as a Contrastive Random Walk
- Contrastive Learning for Space-time Correspondence via Self-cycle Consistency cites Video Object Segmentation Using Space-Time Memory Networks
- Contrastive Learning for Space-time Correspondence via Self-cycle Consistency cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Modulated Contrast for Versatile Image Synthesis cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Neural Matching Fields: Implicit Representation of Matching Fields for Visual Correspondence cites Space-Time Correspondence as a Contrastive Random Walk
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels cites Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels cites A Simple Framework for Contrastive Learning of Visual Representations
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels cites Space-Time Correspondence as a Contrastive Random Walk
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels cites Contrastive Learning for Space-time Correspondence via Self-cycle Consistency
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels cites Video Object Segmentation Using Space-Time Memory Networks
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph Kernels cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
Cited by top-tier papers14
- Modulated Contrast for Versatile Image SynthesisFangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu et al.CVPR 2022 · 44 citations
- Locality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence LearningLiulei Li, Tianfei Zhou, Wenguan Wang, Lu Yang et al.CVPR 2022 · 41 citations
- Neural Matching Fields: Implicit Representation of Matching Fields for Visual CorrespondenceSunghwan Hong, Jisu Nam, Seokju Cho, Susung Hong et al.NeurIPS 2022 · 36 citations
- Exposing the Self-Supervised Space-Time Correspondence Learning via Graph KernelsZheyun Qin, Xiankai Lu, Xiushan Nie, Yilong Yin et al.AAAI 2023 · 22 citations
- Contrastive Learning for Space-time Correspondence via Self-cycle ConsistencyJeany SonCVPR 2022 · 13 citations
Builds on8
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Video Object Segmentation Using Space-Time Memory NetworksSeoung Wug Oh, Joon-Young Lee, Ning Xu, Seon Joo KimICCV 2019 · 845 citations
- Space-Time Correspondence as a Contrastive Random WalkAllan Jabri, Andrew Owens, Alexei A. EfrosNeurIPS 2020 · 356 citations
- SuperGlue: Learning Feature Matching With Graph Neural NetworksPaul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, Andrew RabinovichCVPR 2020
Related papers
- Modelling Neighbor Relation in Joint Space-Time Graph for Video Correspondence LearningZixu Zhao, Yueming Jin, Pheng-Ann HengICCV 2021 · 23 citations
- Spatial-then-Temporal Self-Supervised Learning for Video CorrespondenceRui Li, Dong LiuCVPR 2023
- Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation LearningZhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao et al.CVPR 2021
- Tracking without Label: Unsupervised Multiple Object Tracking via Contrastive Similarity LearningSha Meng, Dian Shao, Jiacheng Guo, Shan GaoICCV 2023 · 14 citations
- Localizing Visual Sounds the Hard WayHonglie Chen, Weidi Xie, Triantafyllos Afouras, Arsha Nagrani et al.CVPR 2021
