IFHE: Intermediate-Feature Heterogeneity Enhancement for Image Synthesis in Data-Free Knowledge Distillation

Data-free knowledge distillation (DFKD) explores training a compact student network only by a pre-trained teacher without real data. Prevailing DFKD methods mainly consist of image synthesis and knowledge distillation. The synthesized images are crucial to enhance the student network performance. However, the images synthesized by existing methods cause high homogeneity on intermediate features, incurring undesired distillation performance. To address this problem, we propose the Intermediate-Feature Heterogeneity Enhancement (IFHE) method, which effectively enhances the heterogeneity of synthesized images by minimizing the loss between intermediate features and pre-set labels of the synthesized images Our IFHE outperforms the SOTA results on CIFAR-10/100 datasets of representative networks.