SpeeDiff: Scalable Pixel-Anchored End-to-End Latent Diffusion Model

We present Scalable Pixel-anchored End-to-end Diffusion (SpeeDiff), a latent diffusion method that jointly trains the VAE and the diffusion model from scratch. In principle, joint training allows the diffusion loss gradient to directly guide the VAE encoder, encouraging the formation of a generationfriendly latent space and potentially yielding faster convergence than the conventional two-stage approach with a pretrained frozen VAE. However, a naive end-to-end implementation severely degrades performance, as unrestricted backpropagation of the diffusion loss leads to latent space collapse. Our main technical contribution is a simple yet effective Tweedie Pixel Reconstruction (TPR) loss, which provides additional pixel-level feedback by decoding a predicted clean latent from an intermediate noisy state using Tweedie's formula, thereby alleviating collapse. Furthermore, our method enables jointly scaling a fully transformerbased architecture and enhances representation alignment within the end-to-end framework. Our SpeeDiff-XL model achieves over 140ˆand 61ˆfaster training compared to Vanilla SiT and REPA, respectively, while attaining an FID of 1.50 without guidance on ImageNet 256ˆ256 generation. With a more efficient 32ˆcompressed VAE, our model further reaches an FID of 1.53 without guidance on ImageNet 512ˆ512 generation.