Diffusion Curriculum: Synthetic-to-Real Data Curriculum via Image-Guided Diffusion

Low-quality or scarce data has posed significant challenges for training deep neural networks in practice. While classical data augmentation cannot produce very different new data, diffusion models open up a new door to build selfevolving AI by generating high-quality and diverse synthetic data through text-guided prompts. However, text-only guidance cannot control synthetic images' proximity to the original images, resulting in out-of-distribution data detrimental to model performance. To overcome the limitation, we study image guidance to achieve a spectrum of interpolations between synthetic and real images. With stronger image guidance, the generated images are similar to the training data, but are hard to learn. With weaker image guidance, the synthetic images will be easier to learn but suffer from a larger distribution gap to the original data. The generated full spectrum of data enables us to build a novel “Diffusion CurricuLum (DisCL)”. DisCL adjusts the image guidance level of image synthesis for each training stage: It identifies and focuses on hard samples for the model and assesses the most effective guidance level of synthetic images to improve hard data learning. We apply DisCL to two challenging tasks: long-tail (LT) classification and learning from lowquality data. It focuses on lower-guidance images of high quality to learn prototypical features as a warm-up for learning higher-guidance images that might be weak on diversity or quality. DisCL achieves a gain of 2.7 % and 2.1 % in OOD and ID macro-accuracy when applied to iWildCam dataset. On ImageNet-LT, DisCL improves the base model's tail-class accuracy from 4.4 % to 23.64 % and leads to a 4.02 % improvement in all-class accuracy.