Lune

CVPR2026Top-tier venue

Guiding a Diffusion Model by Swapping Its Tokens

Weijia Zhang, Yuehao Liu, Shanyan Guan, Wu Ran, Yanhao Ge, Wei Li, Chao Ma

Abstract

Classifier-Free Guidance (CFG) is a widely used inferencetime technique to boost the image quality of diffusion models. Yet, its reliance on text conditions prevents its use in unconditional generation. We propose a simple method to enable CFG-like guidance for both conditional and unconditional generation. The key idea is to generate a perturbed prediction via simple token swap operations, and use the direction between it and the clean prediction to steer sampling towards higher-fidelity distributions. In practice, we swap pairs of most semantically dissimilar token latents in either spatial or channel dimensions. Unlike existing methods that apply perturbation in a global or less constrained manner, our approach selectively exchanges and recomposes token latents, allowing finer control over perturbation and its influence on generated samples. Experiments on MS-COCO 2014, MS-COCO 2017, and ImageNet datasets demonstrate that the proposed Self-Swap Guidance (SSG), when applied to popular diffusion models, outperforms previous condition-free methods in image fidelity and prompt alignment under different set-ups. Its fine-grained perturbation granularity also improves robustness, reducing side-effects across a wider range of perturbation strengths. Overall, SSG extends CFG to a broader scope of applications including both conditional and unconditional generation, and can be readily inserted into any diffusion model as a plug-in to gain immediate improvements. The code is available at https://github.com/VISION-SJTU/SSG.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
  • Diffusion Models Beat GANs on Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
  • Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Implicit Models
  • TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites High-Resolution Image Synthesis with Latent Diffusion Models
  • TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites Diffusion Models Beat GANs on Image Synthesis
  • TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites Denoising Diffusion Probabilistic Models
  • TAG: Tangential Amplifying Guidance for Hallucination-Resistant Sampling cites Denoising Diffusion Implicit Models
  • Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. TAG: Tangential Amplifying Guidance for Hallucination-Resistant SamplingHyunmin Cho, Donghoon Ahn, Susung Hong, Jee Eun Kim et al.ICML 2026

Builds on38

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  4. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  5. Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations

Related papers

  1. Token Perturbation Guidance for Diffusion ModelsJavad Rajabi, Soroush Mehraban, Seyedmorteza Sadat, Babak TaatiNeurIPS 2025 · 17 citations
  2. No Training, No Problem: Rethinking Classifier-Free Guidance for Diffusion ModelsSeyedmorteza Sadat, Manuel Kansy, Otmar Hilliges, Romann M. WeberICLR 2025
  3. Guiding a Diffusion Model with a Bad Version of ItselfTero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen et al.NeurIPS 2024 · 338 citations
  4. Guiding Diffusion Models with Semantically Degraded ConditionsShilong Han, Yuming Zhang, Hongxia WangCVPR 2026 · 1 citation
  5. Improving Classifier-Free Guidance in Masked Diffusion: Low-Dim Theoretical Insights with High-Dim ImpactKevin Rojas, Ye He, Chieh-Hsin Lai, Yuhta Takida et al.ICLR 2026 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines