End-to-End Diffusion Latent Optimization Improves Classifier Guidance

Classifier guidance-using the gradients of an image classifier to steer the generations of a diffusion modelhas the potential to dramatically expand the creative control over image generation and editing. However, currently classifier guidance requires either training new noiseaware models to obtain accurate gradients or using a onestep denoising approximation of the final generation, which leads to misaligned gradients and sub-optimal control.We highlight this approximation's shortcomings and propose a novel guidance method: Direct Optimization of Diffusion Latents (DOODL), which enables plug-and-play guidance by optimizing diffusion latents w.r.t. the gradients of a pre-trained classifier on the true generated pixels, using an invertible diffusion process to achieve memory-efficient backpropagation. Showcasing the potential of more precise guidance, DOODL outperforms one-step classifier guidance on computational and human evaluation metrics across different forms of guidance: using CLIP guidance to improve generations of complex prompts from DrawBench, using fine-grained visual classifiers to expand the vocabulary of Stable Diffusion, enabling image-conditioned generation with a CLIP visual encoder, and improving image aesthetics using an aesthetic scoring network. Code at https://github.com/salesforce/DOODL .