Self-Joint Supervised Learning
Navid Kardan, Mubarak Shah, Mitch Hill
Abstract
Supervised learning is a fundamental framework used to train machine learning systems. A supervised learning problem is often formulated using an i.i.d. assumption that restricts model attention to a single relevant signal at a time when predicting. This contrasts with the human ability to actively use related samples as reference when making decisions. We hypothesize that the restriction to a single signal for each prediction in the standard i.i.d. framework contributes to wellknown drawbacks of supervised learning: making overconfident predictions and vulnerability to overfitting, adversarial attacks, and out-of-distribution data. To address these limitations, we propose a new supervised learning paradigm called self-joint learning that generalizes the standard approach by modeling the joint conditional distribution of two observed samples, where each sample is an image and its label. Rather than assuming samples are independent, our models explicitly learn the sample-to-sample relation of conditional independence. Our framework can naturally incorporate auxiliary unlabeled data to further improve the performance. Experiments on benchmark image datasets show our method offers significant improvement over standard supervised learning in terms of accuracy, robustness against adversarial attacks, out-of-distribution detection, and overconfidence mitigation. Code: github.com/ndkn/Self-joint-Learning
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Exponential Moving Average Normalization for Self-Supervised and Semi-Supervised Learning cites Data-Efficient Image Recognition with Contrastive Predictive Coding
- Exponential Moving Average Normalization for Self-Supervised and Semi-Supervised Learning cites Big Self-Supervised Models are Strong Semi-Supervised Learners
- Exponential Moving Average Normalization for Self-Supervised and Semi-Supervised Learning cites Prototypical Contrastive Learning of Unsupervised Representations
- Big Self-Supervised Models are Strong Semi-Supervised Learners cites Data-Efficient Image Recognition with Contrastive Predictive Coding
- Big Self-Supervised Models are Strong Semi-Supervised Learners cites Prototypical Contrastive Learning of Unsupervised Representations
- Prototypical Contrastive Learning of Unsupervised Representations cites Invariant Information Clustering for Unsupervised Image Classification and Segmentation
Cited by top-tier papers1
Builds on5
- Big Self-Supervised Models are Strong Semi-Supervised LearnersTing Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi et al.NeurIPS 2020 · 2,611 citations
- Data-Efficient Image Recognition with Contrastive Predictive CodingOlivier J. HénaffICML 2020 · 1,553 citations
- Invariant Information Clustering for Unsupervised Image Classification and SegmentationXu Ji, Andrea Vedaldi, João F. HenriquesICCV 2019 · 956 citations
- Prototypical Contrastive Learning of Unsupervised RepresentationsJunnan Li, Pan Zhou, Caiming Xiong, Steven C. H. HoiICLR 2021 · 484 citations
- Exponential Moving Average Normalization for Self-Supervised and Semi-Supervised LearningZhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes et al.CVPR 2021
Related papers
- Rethinking Independent Cross-Entropy Loss For Graph-Structured DataRui Miao, Kaixiong Zhou, Yili Wang, Ninghao Liu et al.ICML 2024 · 5 citations
- Combating Noisy Labels by Agreement: A Joint Training Method with Co-RegularizationHongxin Wei, Lei Feng, Xiangyu Chen, Bo AnCVPR 2020
- Exploring Robustness of Unsupervised Domain Adaptation in Semantic SegmentationJinyu Yang, Chunyuan Li, Weizhi An, Hehuan Ma et al.ICCV 2021 · 34 citations
- Self-supervised Label Augmentation via Input TransformationsHankook Lee, Sung Ju Hwang, Jinwoo ShinICML 2020 · 218 citations
- Can Generative Models Improve Self-Supervised Representation Learning?Sana Ayromlou, Vahid Reza Khazaie, Fereshteh Forghani, Arash AfkanpourAAAI 2025 · 5 citations
