GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost

Recent advancements in dataset distillation have demonstrated the significant benefits of employing soft labels generated by pre-trained teacher models. In this paper, we introduce a novel perspective by emphasizing the full utilization of labels. We first conduct a comprehensive comparison of various loss functions for soft label utilization in dataset distillation, revealing that the model trained on the synthetic dataset exhibits high sensitivity to the choice of loss function for soft label utilization. This finding highlights the necessity of a universal loss function for training models on synthetic datasets. Building on these insights, we introduce an extremely simple yet surprisingly effective plug-and-play approach, GIFT, which encompasses soft label refinement and a cosine similarity-based loss function to efficiently leverage full label information. Extensive experiments demonstrate that GIFT consistently enhances the state-of-the-art dataset distillation methods across various scales datasets without incurring additional computational costs. For instance, on ImageNet-1K with IPC = 10, GIFT improves the SOTA method RDED by 3.9% and 1.8% on ConvNet and ResNet-18, respectively. Code: https://github.com/LINs-lab/GIFT . 3 * Equal contribution with the first author † Corresponding author. 3 We have provided Pytorch implementation code in Appendix E 4 The synthetic dataset of DATM for ImageNet-1k is not provided, so comparing with DATM is not possible on this dataset 5 Detailed experiments are elaborated in Section 3. The results of Tiny-ImageNet and ImageNet-1K when IPC = 1 and IPC = 50 are shown in Appendix D Preprint. Under review.