Lune

CVPR2023Top-tier venue

A Data-Based Perspective on Transfer Learning

Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, Aleksander Madry

Abstract

It is commonly believed that in transfer learning including more pre-training data translates into better performance. However, recent evidence suggests that removing data from the source dataset can actually help too. In this work, we take a closer look at the role of the source dataset's composition in transfer learning and present a framework for probing its impact on downstream performance. Our framework gives rise to new capabilities such as pinpointing transfer learning brittleness as well as detecting pathologies such as data-leakage and the presence of misleading examples in the source dataset. In particular, we demonstrate that removing detrimental datapoints identified by our framework indeed improves transfer learning performance from ImageNet on a variety of target tasks. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning cites What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
  • Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning cites Do Adversarially Robust ImageNet Models Transfer Better?
  • On the Robustness of Vision Transformers to Adversarial Examples cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • DataMIL: Selecting Data for Robot Imitation Learning with Datamodels cites What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation
  • The Value of Out-of-Distribution Data cites Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. DataMIL: Selecting Data for Robot Imitation Learning with DatamodelsShivin Dass, Alaa Khaddaj, Logan Engstrom, Aleksander Madry et al.ICLR 2026 · 34 citations
  2. Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer LearningYihua Zhang, Yimeng Zhang, Aochuan Chen, Jinghan Jia et al.NeurIPS 2023 · 18 citations
  3. The Value of Out-of-Distribution DataAshwin De Silva, Rahul Ramesh, Carey E. Priebe, Pratik Chaudhari et al.ICML 2023 · 16 citations
  4. Conditional Language Learning with ContextXiao Zhang, Miao Li, Ji WuICML 2024 · 5 citations
  5. MLAS-LoRA: Language-Aware Parameters Detection and LoRA-Based Knowledge Transfer for Multilingual Machine TranslationTianyu Dong, Bo Li, Jinsong Liu, Shaolin Zhu et al.ACL 2025

Builds on6

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-DistributionAnanya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma et al.ICLR 2022 · 911 citations
  3. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence EstimationVitaly Feldman, Chiyuan ZhangNeurIPS 2020 · 674 citations
  4. Do Adversarially Robust ImageNet Models Transfer Better?Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor et al.NeurIPS 2020 · 506 citations
  5. On the Robustness of Vision Transformers to Adversarial ExamplesKaleel Mahmood, Rigel Mahmood, Marten van DijkICCV 2021 · 261 citations

Related papers

  1. Adversarial Training Helps Transfer Learning via Better RepresentationsZhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi et al.NeurIPS 2021 · 60 citations
  2. Retaining Beneficial Information from Detrimental Data for Neural Network RepairLong-Kai Huang, Peilin Zhao, Junzhou Huang, Sinno Jialin PanNeurIPS 2023 · 2 citations
  3. Investigating Transferability in Pretrained Language ModelsAlex Tamkin, Trisha Singh, Davide Giovanardi, Noah D. GoodmanEMNLP Findings 2020 · 45 citations
  4. Adversarially-Trained Deep Nets Transfer Better: Illustration on Image ClassificationFrancisco Utrera, Evan Kravitz, N. Benjamin Erichson, Rajiv Khanna et al.ICLR 2021 · 42 citations
  5. TransTailor: Pruning the Pre-trained Model for Improved Transfer LearningBingyan Liu, Yifeng Cai, Yao Guo, Xiangqun ChenAAAI 2021 · 69 citations
Dusk over the sea between two cliffs drawn in fine vertical lines