Lune

CVPR Findings2026

Bootstrap Your Own Classifier: Your Pretrained Vision Models are Secretly Strong Continual Learners

Yizheng Gong, Xiaoyang Wang, Siyue Yu, Waleed Al-Nuaimy, Jimin Xiao

Abstract

Continual learning requires models to sequentially acquire new knowledge while preserving previously learned information. Current approaches combine pretrained vision backbones with randomly initialized classifiers, creating an initialization asymmetry that limits performance. We propose Bootstrap Your Own Classifier (BYOC), which initializes classifiers using class prototypes extracted from the pretrained backbone, establishing a fully coherent initialization. This simple change dramatically stabilizes sequential fine-tuning. To address the gradual drift between the evolving backbone and previously learned classifiers, we employ exponential moving average updates for the backbone and calibrate historical classifiers based on current task adaptation patterns. Despite its simplicity, BYOC achieves state-of-the-art performance on CIFAR-100, ImageNet-R, ImageNet-A, CUB-200, and Stanford Cars, demonstrating that proper initialization from pretrained representations substantially outperforms complex parameter-efficient methods. Our work reveals that pretrained vision models are inherently strong continual learners when provided with consistent initialization across all components. Code is available at: https://github.com/jordangong/ continual-learning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Emerging Properties in Self-Supervised Vision Transformers cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Emerging Properties in Self-Supervised Vision Transformers cites Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on24

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Bootstrap Your Own Latent - A New Approach to Self-Supervised LearningJean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec et al.NeurIPS 2020 · 9,171 citations
  4. Emerging Properties in Self-Supervised Vision TransformersMathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou et al.ICCV 2021 · 8,921 citations
  5. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationDan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath et al.ICCV 2021 · 2,294 citations

Related papers

  1. Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformerYuwen Tan, Qinhao Zhou, Xiang Xiang, Ke Wang et al.CVPR 2024 · 14 citations
  2. Adapt Before Continual LearningAojun Lu, Tao Feng, Hangjie Yuan, Chunhui Ding et al.AAAI 2026
  3. RanPAC: Random Projections and Pre-trained Models for Continual LearningMark D. McDonnell, Dong Gong, Amin Parvaneh, Ehsan Abbasnejad et al.NeurIPS 2023 · 245 citations
  4. SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained ModelGengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen et al.ICCV 2023 · 196 citations
  5. SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained ModelsLinglan Zhao, Xuerui Zhang, Ke Yan, Shouhong Ding et al.NeurIPS 2024 · 22 citations
Dusk over the sea between two cliffs drawn in fine vertical lines