Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization

As large-scale diffusion models continue to advance, they excel at producing high-quality images but often generate unwanted content, such as sexually explicit or violent imagery. Existing methods for removing such content typically guide the image generation process but can inadvertently modify unrelated regions, leading to inconsistencies with the original model. We propose a novel approach for targeted concept replacement in diffusion models, enabling specific unwanted concepts to be removed without significantly affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, using few-shot learning to minimize the need for labeled data.Within the identified region, we further introduce a Dual Prompts Cross-Attention module that operates without additional training to substitute the target concept, ensuring minimal disruption to surrounding content. We evaluate our method in terms of concept localization precision and replacement efficiency. Experimental results demonstrate that our method achieves superior precision in localizing target concepts and performs coherent concept replacement with minimal impact on non-target areas, outperforming existing approaches. The code is available at https://github.com/zhang-lingyun/ConceptReplacer