ColorDiffuser: Video Colorization with Pretrained Text-to-Image Diffusion Models

Video colorization is inherently challenging due to the need for accurate color inference and temporal consistency. In this paper, we present ColorDiffuser, an adaptation of a pre-trained text-to-image latent diffusion model for video colorization. By leveraging learned color priors from large-scale training, our method avoids costly retraining and enables controllable colorization via text prompts. To address the adaptation of an image model to video, we propose a novel Short- and Long-distance Cross-Frame Attention (SL-CFA) module combined with an amortized sampling strategy to unify the color latent over time. By incorporating information from nearby and distant frames, the model achieves better consistency for long video sequences, even with problematic disocclusion. To mitigate visual detail loss and color bleeding from compressed latent representations, we introduce a video colorization VAE model that incorporates semantic boundaries and grayscale inputs. Extensive experiments on benchmark datasets demonstrate that ColorDiffuser achieves state-of-the-art performance in color fidelity, temporal consistency, and visual quality, while offering diverse and controllable outputs. Our project page can be accessed at: https://colordiffuser.github.io.