What Makes Transfer Learning Work for Medical Images: Feature Reuse & Other Factors

Transfer learning is a standard technique to transfer knowledge from one domain to another. For applications in medical imaging, transfer from ImageNet has become the de-facto approach, despite differences in the tasks and im-age characteristics between the domains. However, it is un-clear what factors determine whether - and to what extent- transfer learning to the medical domain is useful. The long- standing assumption that features from the source domain get reused has recently been called into question. Through a series of experiments on several medical image bench-mark datasets, we explore the relationship between transfer learning, data size, the capacity and inductive bias of the model, as well as the distance between the source and tar-get domain. Our findings suggest that transfer learning is beneficial in most cases, and we characterize the important role feature reuse plays in its success.