Unveiling Concept Attribution in Diffusion Models
Nguyen Hung-Quang, Hoang Phan, Khoa D. Doan
Abstract
Diffusion models have shown remarkable abilities in generating realistic and highquality images from text prompts. However, a trained model remains largely black-box; little do we know about the roles of its components in exhibiting a concept such as objects or styles. Recent works employ causal tracing to localize knowledge-storing layers in generative models without showing how other layers contribute to the target concept. In this work, we approach diffusion models' interpretability problem from a more general perspective and pose a question: "How do model components work jointly to demonstrate knowledge?". To answer this question, we decompose diffusion models using component attribution, systematically unveiling the importance of each component (specifically the model parameter) in generating a concept. The proposed framework, called Component Attribution for Diffusion Model (CAD), discovers the localization of concept-inducing (positive) components, while interestingly uncovers another type of components that contribute negatively to generating a concept, which is missing in the previous knowledge localization work. Based on this holistic understanding of diffusion models, we present and empirically evaluate one utility of component attribution in controlling the generation process. Specifically, we introduce two fast, inference-time model editing algorithms, CAD-Erase and CAD-Amplify; in particular, CAD-Erase enables erasure and CAD-Amplify allows amplification of a generated concept by ablating the positive and negative components, respectively, while retaining knowledge of other concepts. Extensive experimental results validate the significance of both positive and negative components pinpointed by our framework, demonstrating the potential of providing a complete view of interpreting generative models. Our code is available here.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Denoising Diffusion Probabilistic Models
- SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Denoising Diffusion Implicit Models
- Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts cites High-Resolution Image Synthesis with Latent Diffusion Models
- Continual Unlearning for Text-to-Image Diffusion Models: A Regularization Perspective cites High-Resolution Image Synthesis with Latent Diffusion Models
- Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers2
- Continual Unlearning for Text-to-Image Diffusion Models: A Regularization PerspectiveJustin Lee, Zheda Mai, Jinsu Yoo, Chongyu Fan et al.ICLR 2026 · 9 citations
- Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted ConceptsLeyang Li, Shilin Lu, Yan Ren, Adams Wai-Kin KongACM MM 2025 · 4 citations
Builds on27
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
- SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
Related papers
- Concept-TRAK: Understanding how diffusion models learn concepts through concept attributionYong-Hyun Park, Chieh-Hsin Lai, Satoshi Hayakawa, Yuhta Takida et al.ICLR 2026 · 4 citations
- Localizing and Editing Knowledge In Text-to-Image Generative ModelsSamyadeep Basu, Nanxuan Zhao, Vlad I. Morariu, Soheil Feizi et al.ICLR 2024 · 50 citations
- Localized Concept Erasure in Text-to-Image Diffusion Models via High-Level Representation MisdirectionUichan Lee, Jeonghyeon Kim, Sangheum HwangICLR 2026 · 3 citations
- On Mechanistic Knowledge Localization in Text-to-Image Generative ModelsSamyadeep Basu, Keivan Rezaei, Priyatham Kattakinda, Vlad I. Morariu et al.ICML 2024 · 30 citations
- From Inpainting to Layer Decomposition: Repurposing Generative Inpainting Models for Image Layer DecompositionJingxi Chen, Yixiao Zhang, Xiaoye Qian, Zongxia Li et al.CVPR 2026 · 5 citations
