Guiding a Diffusion Model by Swapping Its Tokens
Weijia Zhang, Yuehao Liu, Shanyan Guan, Wu Ran, Yanhao Ge, Wei Li, Chao Ma
Abstract
Classifier-Free Guidance (CFG) is a widely used inferencetime technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements. The code is available at https://github.com/VISION-SJTU/SSG.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
- Diffusion Models Beat GANs on Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Implicit Models
- TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites High-Resolution Image Synthesis with Latent Diffusion Models
- TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites Diffusion Models Beat GANs on Image Synthesis
- TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites Denoising Diffusion Probabilistic Models
- TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites Denoising Diffusion Implicit Models
- Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers1
Builds on38
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
Related papers
- Token Perturbation Guidance for Diffusion ModelsJavad Rajabi, Soroush Mehraban, Seyedmorteza Sadat, Babak TaatiNeurIPS 2025 · 17 citations
- No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion ModelsSeyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, Romann M. WeberICLR 2025
- Guiding a Diffusion Model with a Bad Version of ItselfTero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen et al.NeurIPS 2024 · 338 citations
- Guiding Diffusion Models with Semantically Degraded ConditionsShilong Han, Yuming Zhang, Hongxia WangCVPR 2026 · 1 citation
- Improving Classifier-Free Guidance in Masked Diffusion: Low-Dim Theoretical Insights with High-Dim ImpactKevin Rojas, Ye He, Chieh-Hsin Lai, Yuhta Takida et al.ICLR 2026 · 11 citations
