A Unified Framework for Diffusion Model Unlearning with f-Divergence

Most existing methods for concept unlearning in text-to-image diffusion models minimize a mean squared error (MSE) loss between the denoiser outputs conditioned on a target and an anchor concept, which is implicitly the KL divergence between two Gaussians. We generalize this objective to any f -divergence, recovering MSE as the KL instance, and identify a family of α-divergences whose Gaussian closed-form yields cheap, MSE-like training objectives. For the remaining f -divergences, we provide a minmax objective based on the variational formulation of the f -divergence. We theoretically analyze and numerically validate how different fdivergences impact the gradient magnitude and the convergence properties of the algorithm, affecting the quality of unlearning. For instance, we observe that the Hellinger closed-form instance consistently dominates MSE across multiple scenarios. More generally, the proposed unified framework offers a flexible paradigm for selecting the optimal divergence based on the application and user goal, allowing for finer control over the trade-off between unlearning efficacy and generative fidelity. Code available at https: //github.com/tonellolab/f-DMU. WARNING: This paper contains model outputs that may be offensive.