Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift

Personalizing text-to-image diffusion models involves integrating novel visual concepts from a small set of reference images while retaining the model's original generative capabilities. However, this process often leads to overfitting, where the model ignores the user's prompt and merely replicates the reference images. We attribute this issue to a fundamental misalignment between the true goals of personalization, which are subject fidelity and text alignment, and the training objectives of existing methods that fail to enforce both objectives simultaneously. Specifically, prior approaches often overlook the need to explicitly preserve the pretrained model's output distribution, resulting in distributional drift that undermines diversity and coherence. To resolve these challenges, we introduce a Lipschitz-based regularization objective that constrains parameter updates during personalization, ensuring bounded deviation from the original distribution. This promotes consistency with the pretrained model's behavior while enabling accurate adaptation to new concepts. Furthermore, our method offers a computationally efficient alternative to commonly used, resource-intensive sampling techniques. Through extensive experiments across diverse diffusion model architectures, we demonstrate that our approach achieves superior performance in both quantitative metrics and qualitative evaluations, consistently excelling in visual fidelity and prompt adherence. We further support these findings with comprehensive analyses, including ablation studies and visualizations. † Corresponding author Personalized Text-to-Image Generation. A central question of few-shot personalization has been how to adapt pretrained networks to new concepts with only a few subject-specific images. Textual Inversion (Gal et al., 2022; Voynov et al., 2023) encodes subject-specific information into learned text 1 This follows from the conditional distribution log p θ (x | c) = log p θ (x, c) -log p(c) = log ∫︁ p θ (x, z1:T , c) dz1:T -log p(c), where log p(c) is constant with respect to θ.