Implicit Modeling for Transferability Estimation of Vision Foundation Models

Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning. This capability facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model's intrinsic transferability, coupled with a Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables generalization across a broader range of models and downstream tasks. Extensive experiments on a comprehensive benchmark-spanning extensive training regimes and a wider variety of model types-demonstrate that ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency. Code is available at https://github.com/BUAAHugeGun/ITM.