Unified Structural Factors for Transfer Learning Generalization with PAC-Bayesian Guarantees
Ziqi Gao
Abstract
Understanding when a pre-trained model generalizes well to a new task remains a key challenge in transfer learning. Classical theories bound target risk using divergences such as total variation, MMD, or Wasserstein distance, yet tasks with similar divergences often show very different transfer performance. We propose a structural framework that explains transferability through two factors: the Feature Overlap Rate (FOR), measuring how much target representation lies in the source-induced subspace, and the Effective Task Complexity (ETC), quantifying the entropy of latent subtasks. We derive a PAC-Bayesian bound where target risk depends on FOR and ETC, and show that larger models attenuate their negative effects. Experiments on six GLUE transfer pairs estimate FOR and ETC from encoder representations and compare them to classical divergences. Results show that FOR and ETC together explain over 80% of transfer risk variance, while divergences fail to do so. Our findings provide a geometry-aware perspective for diagnosing and guiding transfer learning.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Builds on5
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Contrastive Representation DistillationYonglong Tian, Dilip Krishnan, Phillip IsolaICLR 2020 · 1,305 citations
- LEEP: A New Measure to Evaluate Transferability of Learned RepresentationsCuong V. Nguyen, Tal Hassner, Matthias W. Seeger, Cédric ArchambeauICML 2020 · 279 citations
- Invariant RationalizationShiyu Chang, Yang Zhang, Mo Yu, Tommi S. JaakkolaICML 2020 · 232 citations
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningArmen Aghajanyan, Sonal Gupta, Luke ZettlemoyerACL 2021
Related papers
- Frustratingly Easy Transferability EstimationLong-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang et al.ICML 2022 · 71 citations
- Understanding the Transferability of Representations via Task-RelatednessAkshay Mehra, Yunbei Zhang, Jihun HammNeurIPS 2024 · 13 citations
- OTCE: A Transferability Metric for Cross-Domain Cross-Task RepresentationsYang Tan, Yang Li, Shao-Lun HuangCVPR 2021
- The Impact of Geometric Complexity on Neural Collapse in Transfer LearningMichael Munn, Benoit Dherin, Javier GonzalvoNeurIPS 2024 · 6 citations
- Transferability Estimation using Bhattacharyya Class SeparabilityMichal Pándy, Andrea Agostinelli, Jasper R. R. Uijlings, Vittorio Ferrari et al.CVPR 2022 · 50 citations
