Exploring Incompatible Knowledge Transfer in Few-shot Image Generation

ProgressiveGAN StyleGAN-V2 Church ↦ Sailboat FFHQ ↦ AFHQ-Cat Church ↦ Sailboat FFHQ ↦ AFHQ-Cat Pretraining on large source dataset (e.g., LSUN-Church, FFHQ…) Adaptation on few-shot target samples (e.g., Sailboat, AFHQ-Cat…) G s G t SailBoat (10-shot) initialize LSUN-Church (126 K) Resulting generator Transfer Learning for FSIG TGAN AdAM Ours EWC AFHQ-Cat (10-shot) SailBoat (10-shot) Resulting generator (a) (c) G s (b) Figure 1. In FSIG, transfer of incompatible knowledge would cause significant degradation of the realisticness of synthetic samples, and existing SOTA methods fail to address this issue. (a) Prior knowledge of the source GAN generator Gs is selected, preserved, and transferred to learn the target generator Gt, given very limited target domain training samples (see 10-shot in (b)). In this work, we consider challenging setups where the target domain could be semantically distant from the source domain, e.g., (i) Church Ñ Sailboat (ii) FFHQ Ñ AFHQ-Cat. (c) Synthetic images from Gs, TGAN [65]; and SOTA methods: EWC [41] and AdAM [75]. Images in each column are from the same noise input. Critically, existing FSIG methods focus on knowledge preservation, while failing to prevent the transfer of knowledge that is incompatible with the target domain. This gives rise to unrealistic samples generated by Gt, e.g., "Trees/Buildings on the Sea" (Red frames), or "Cats with Glasses" (Green frames). In contrast, our method (row-5 in (c)) based on knowledge truncation can effectively prevent the transfer of incompatible knowledge. See our detailed analysis in Sec. 4. Best viewed in color with zooming in.