Forget Many, Forget Right: Scalable and Precise Concept Unlearning in Diffusion Models
Kaiyuan Deng, Gen Li, Yang Xiao, Bo Hui, Xiaolong Ma
Abstract
While multi-concept unlearning has shown progress, extending to large-scale scenarios remains difficult, as existing methods face three persistent challenges: (i) they often introduce conflicting weight updates, making some targets difficult to unlearn or causing degradation of generative capability; (ii) they lack precise mechanisms to keep unlearning strictly confined to target concepts, resulting in collateral damage on similar content; (iii) many approaches rely on additional data or auxiliary modules, causing scalability and efficiency bottlenecks as the number of concepts grows. To simultaneously address these challenges, we propose Scalable-Precise Concept Unlearning (ScaPre), a unified and lightweight framework tailored for scalable and precise large-scale unlearning. ScaPre introduces a conflict-aware stable design, which integrates the spectral trace regularizer and geometry alignment to stabilize the optimization space, suppress conflicting updates, and preserve the pretrained global structure. Furthermore, the Informax Decoupler identifies concept-relevant parameters and adaptively reweights updates, ensuring that unlearning is confined to the target subspace without collateral damage. ScaPre yields an efficient closed-form solution, requiring no additional data or auxiliary sub-models, while maintaining both scalability and precision. Comprehensive experiments across large-scale objects, styles, and explicit content benchmarks demonstrate that ScaPre effectively removes target concepts while maintaining generation quality. It can forget up to ×5 more concepts than the best baseline within the limits of acceptable generative quality, and outperforms existing multi-concept approaches in precision and efficiency, achieving a new state of the art for large-scale unlearning.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 7378974b-c202-4472-9dc7-50211c953df3Cited by top-tier papers3
- SAEmnesia: Erasing Concepts in Diffusion Models with Supervised Sparse AutoencodersEnrico Cassano, Riccardo Renzulli, Marco Nurisso, Mirko Zaffaroni et al.ICML 2026 · 7 citations
- Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron MaskingKaiyuan Deng, Bo Hui, Gen Li, Jie Ji et al.ICML 2026 · 1 citation
- UnHype: CLIP-Guided Hypernetworks for Dynamic LoRA UnlearningPiotr Wójcik, Maksym Petrenko, Wojciech Gromski, Przemysław Spurek et al.ICML 2026 · 1 citation
Builds on24
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion ModelsAlexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam et al.ICML 2022 · 4,691 citations
- SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations
- Machine UnlearningLucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia et al.S&P 2021 · 1,381 citations
Related papers
- Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware OptimizationGen Li, Yang Xiao, Jie Ji, Kaiyuan Deng et al.ICCV 2025 · 1 citation
- SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse AutoencodersJiahui Geng, Qing LiICCV 2025 · 12 citations
- Boosting Alignment for Post-Unlearning Text-to-Image Generative ModelsMyeongseob Ko, Henry Li, Zhun Wang, Jonathan Patsenker et al.NeurIPS 2024 · 22 citations
- Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric TasksYu Zhou, Dian Zheng, Qijie Mo, Renjie Lu et al.CVPR 2025
- Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned ConceptsHongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu et al.ICCV 2025 · 4 citations
