Unified Structural Factors for Transfer Learning Generalization with PAC-Bayesian Guarantees
Ziqi Gao
摘要
Understanding when a pre-trained model generalizes well to a new task remains a key challenge in transfer learning. Classical theories bound target risk using divergences such as total variation, MMD, or Wasserstein distance, yet tasks with similar divergences often show very different transfer performance. We propose a structural framework that explains transferability through two factors: the Feature Overlap Rate (FOR), measuring how much target representation lies in the source-induced subspace, and the Effective Task Complexity (ETC), quantifying the entropy of latent subtasks. We derive a PAC-Bayesian bound where target risk depends on FOR and ETC, and show that larger models attenuate their negative effects. Experiments on six GLUE transfer pairs estimate FOR and ETC from encoder representations and compare them to classical divergences. Results show that FOR and ETC together explain over 80% of transfer risk variance, while divergences fail to do so. Our findings provide a geometry-aware perspective for diagnosing and guiding transfer learning.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper5
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh 等ICML 2021 · 被引用 47,906 次
- Contrastive Representation DistillationYonglong Tian, Dilip Krishnan, Phillip IsolaICLR 2020 · 被引用 1,305 次
- LEEP: A New Measure to Evaluate Transferability of Learned RepresentationsCuong V. Nguyen, Tal Hassner, Matthias W. Seeger, Cédric ArchambeauICML 2020 · 被引用 279 次
- Invariant RationalizationShiyu Chang, Yang Zhang, Mo Yu, Tommi S. JaakkolaICML 2020 · 被引用 232 次
- Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningArmen Aghajanyan, Sonal Gupta, Luke ZettlemoyerACL 2021
相关 Paper
- Frustratingly Easy Transferability EstimationLong-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang 等ICML 2022 · 被引用 71 次
- Understanding the Transferability of Representations via Task-RelatednessAkshay Mehra, Yunbei Zhang, Jihun HammNeurIPS 2024 · 被引用 13 次
- OTCE: A Transferability Metric for Cross-Domain Cross-Task RepresentationsYang Tan, Yang Li, Shao-Lun HuangCVPR 2021
- The Impact of Geometric Complexity on Neural Collapse in Transfer LearningMichael Munn, Benoit Dherin, Javier GonzalvoNeurIPS 2024 · 被引用 6 次
- Transferability Estimation using Bhattacharyya Class SeparabilityMichal Pándy, Andrea Agostinelli, Jasper R. R. Uijlings, Vittorio Ferrari 等CVPR 2022 · 被引用 50 次
