Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models
Taj Jones-McCormick, Aukosh Jagannath, Subhabrata Sen
Abstract
Unsupervised pre-training and transfer learning are commonly used techniques to initialize training algorithms for neural networks, particularly in settings with limited labeled data. In this paper, we study the effects of unsupervised pre-training and transfer learning on the sample complexity of high-dimensional supervised learning. Specifically, we consider the problem of training a singlelayer neural network via online stochastic gradient descent. We establish that pre-training and transfer learning (under concept shift) reduce sample complexity by polynomial factors (in the dimension) under very general assumptions. We also uncover some surprising settings where pre-training grants exponential improvement over random initialization in terms of sample complexity.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on15
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Domain Adaptation with Conditional Distribution Matching and Generalized Label ShiftRemi Tachet des Combes, Han Zhao, Yu-Xiang Wang, Geoffrey J. GordonNeurIPS 2020 · 231 citations
- Predicting What You Already Know Helps: Provable Self-Supervised LearningJason D. Lee, Qi Lei, Nikunj Saunshi, Jiacheng ZhuoNeurIPS 2021 · 219 citations
- Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt TuningColin Wei, Sang Michael Xie, Tengyu MaNeurIPS 2021 · 119 citations
- Phase retrieval in high dimensions: Statistical and computational phase transitionsAntoine Maillard, Bruno Loureiro, Florent Krzakala, Lenka ZdeborováNeurIPS 2020 · 73 citations
Related papers
- The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate ShiftJingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu et al.NeurIPS 2022 · 29 citations
- What Do Neural Networks Learn When Trained With Random Labels?Hartmut Maennel, Ibrahim M. Alabdulmohsin, Ilya O. Tolstikhin, Robert J. N. Baldock et al.NeurIPS 2020 · 99 citations
- Transfer Learning in Infinite Width Feature Learning NetworksClarissa Lauditi, Blake Bordelon, Cengiz PehlevanICLR 2026 · 2 citations
- Unsupervised Learning for Combinatorial Optimization Needs Meta LearningHaoyu Peter Wang, Pan LiICLR 2023 · 2 citations
- On the Provable Advantage of Unsupervised PretrainingJiawei Ge, Shange Tang, Jianqing Fan, Chi JinICLR 2024 · 23 citations
