State Representation Learning Using an Unbalanced Atlas
Abstract
The manifold hypothesis posits that high-dimensional data often lies on a lowerdimensional manifold and that utilizing this manifold as the target space yields more efficient representations. While numerous traditional manifold-based techniques exist for dimensionality reduction, their application in self-supervised learning has witnessed slow progress. The recent MSimCLR method combines manifold encoding with SimCLR but requires extremely low target encoding dimensions to outperform SimCLR, limiting its applicability. This paper introduces a novel learning paradigm using an unbalanced atlas (UA), capable of surpassing state-of-the-art self-supervised learning approaches 1 . We investigated and engineered the DeepInfomax with an unbalanced atlas (DIM-UA) method by adapting the Spatiotemporal DeepInfomax (ST-DIM) framework to align with our proposed UA paradigm. The efficacy of DIM-UA is demonstrated through training and evaluation on the Atari Annotated RAM Interface (AtariARI) benchmark, a modified version of the Atari 2600 framework that produces annotated image samples for representation learning. The UA paradigm improves existing algorithms significantly as the number of target encoding dimensions grows. For instance, the mean F1 score averaged over categories of DIM-UA is ∼75% compared to ∼70% of ST-DIM when using 16384 hidden units.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- A Simple Framework for Contrastive Learning of Visual Representations cites Momentum Contrast for Unsupervised Visual Representation Learning
- Momentum Contrast for Unsupervised Visual Representation Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Exploring Simple Siamese Representation Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Exploring Simple Siamese Representation Learning cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
- Exploring Simple Siamese Representation Learning cites Momentum Contrast for Unsupervised Visual Representation Learning
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites A Simple Framework for Contrastive Learning of Visual Representations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning cites Momentum Contrast for Unsupervised Visual Representation Learning
Builds on4
- A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
- Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
- Exploring Simple Siamese Representation LearningXinlei Chen, Kaiming HeCVPR 2021
- Momentum Contrast for Unsupervised Visual Representation LearningKaiming He, Haoqi Fan, Yuxin Wu, Saining Xie et al.CVPR 2020
Related papers
- On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement LearningHojoon Lee, Koanho Lee, Dongyoon Hwang, Hyunho Lee et al.ICML 2023 · 11 citations
- Efficient Distribution Matching of Representations via Noise-Injected Deep InfoMaxIvan Butakov, Alexander Semenenko, Alexander Tolmachev, Andrey Gladkov et al.ICLR 2025
- Self-labelling via simultaneous clustering and representation learningYuki Markus Asano, Christian Rupprecht, Andrea VedaldiICLR 2020 · 873 citations
- Understanding Dimensional Collapse in Contrastive Self-supervised LearningLi Jing, Pascal Vincent, Yann LeCun, Yuandong TianICLR 2022 · 467 citations
- Self-Supervised Set Representation Learning for Unsupervised Meta-LearningDong Bok Lee, Seanie Lee, Kenji Kawaguchi, Yunji Kim et al.ICLR 2023
