Boosting Latent Diffusion with Perceptual Objectives

Latent diffusion models (LDMs) (Rombach et al., 2022) have enabled considerable advances in image generation, and elevated the problem of generative image modeling to a level where it has become available as a technology to the public. A critical part to this success is to define the generative model in the latent space of an autoencoder (AE), which reduces the resolution of the representation over which the model is defined, thereby making it possible to scale diffusion methods to larger datasets, resolutions, and architectures than original pixel-based diffusion models (Dhariwal & Nichol, 2021; Sohl-Dickstein et al., 2015) . To train an LDM, all images are first projected into a latent space with the encoder of a pre-trained autoencoder, and then, the diffusion model is optimized directly in the latent space. Note that when learning the diffusion model the AE decoder is not used -the diffusion model does not receive any training feedback that would ensure that all latent values reachable by the diffusion process decode to a high quality image. This training procedure leads to a disconnect between the diffusion model and the AE decoder, prompting the LDM to produce low quality images that oftentimes lack high frequency image components. Moreover, we note that the latent spaces of pre-trained LDM's autoencoders tend to be highly irregular, in the sense that small changes in the latent space can lead to large changes in the generated images, further exacerbating the autoencoder-diffusion disconnect problem. In this work, we propose to alleviate this autoencoder-diffusion disconnect and propose to include the AE decoder in the training objective of LDM. In particular, we introduce latent perceptual loss (LPL) that acts on the decoder's intermediate features to enrich the training signal of LDM. This is similar to the use of perceptual losses for image-to-image translation tasks (Johnson et al., 2016; Zhang et al., 2018) , but we apply this idea in the context of generative modeling and use the feature space of the pre-trained AE decoder rather than that of an external pre-trained discriminative network. Our latent perceptual loss results in sharper and more realistic images, and leads to better structural consistency than the baseline -see Figure 1 . We validate LPL on three datasets of different sizes -the commonly used datasets ImageNet-1k (1M data points) and CC12M (12M data points), and additionally a private dataset S320M (320M data points) -as well as three generative models formulation -DDPM (Ho et al., 2020) with velocity and epsilon prediction, and conditional flow matching model (Lipman et al., 2023) . In our experiments, we report standard image generative model metrics -such as FID (Heusel et al., 2017) , CLIPScore (Hessel et al., 2021) , as well as Precision and Recall (Sajjadi et al., 2018; Kynkäänniemi et al., 2019) . Our experiments show that the use of LPL leads to consistent performance boosts between 6% and 20% in terms of FID. Our qualitative analysis further highlights the benefits of LPL, showing images that are sharp and contain high-frequency image details. In summary, our contributions are: • We identify a slight disconnect between latent and pixel-space diffusion which can lead to suboptimal results when training latent diffusion and flow models. • We propose the latent perceptual loss (LPL), a perceptual loss variant leveraging the intermediate feature representation of the autoencoder's decoder. • We present extensive experimental results on the ImageNet-1k, CC12M, and S320M datasets, demonstrating the benefits of LPL in boosting the model's quality by 6% to 20% in terms of FID. • We show that LPL is effective for a variety of generative model formulations including DDPM and conditional flow matching approaches. Related work Diffusion models. The generative modeling landscape has been significantly impacted by diffusion models, surpassing previous state-of-the-art GAN-based methods (