Training-free Dense-Aligned Diffusion Guidance for Modular Conditional Image Synthesis
Abstract
Conditional image synthesis is a crucial task with broad applications, such as artistic creation and virtual reality. However, current generative methods are often taskoriented with a narrow scope, handling a restricted condition with constrained applicability. In this paper, we propose a novel approach that treats conditional image synthesis as the modular combination of diverse fundamental condition units. Specifically, we divide conditions into three primary units: text, layout, and drag. To enable effective control over these conditions, we design a dedicated alignment module for each. For the text condition, we introduce a Dense Concept Alignment (DCA) module, which achieves dense visual-text alignment by drawing on diverse textual concepts. For the layout condition, we propose a Dense Geometry Alignment (DGA) module to enforce comprehensive geometric constraints that preserve the spatial configuration. For the drag condition, we introduce a Dense Motion Alignment (DMA) module to apply multi-level motion regularization, ensuring that each pixel follows its desired trajectory without visual artifacts. By flexibly inserting and combining these alignment modules, our framework enhances the model's adaptability to diverse conditional generation tasks and greatly expands its application range. Extensive experiments demonstrate the superior performance of our framework across a variety of conditions, including textual description, segmentation mask (bounding box), drag manipulation, and their combinations. Code is available at https://github.com/ZixuanWang0525/DADG
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
- High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Implicit Models
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Probabilistic Models
- Diffusion Models Beat GANs on Image Synthesis cites Denoising Diffusion Implicit Models
- Dual Optimal Transport for Multi-Concept Composition: Structural Alignment and Texture Injection in Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
- Denoising Diffusion Implicit Models cites Denoising Diffusion Probabilistic Models
Cited by top-tier papers1
Builds on33
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
- Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
Related papers
- Text-Conditional Attribute Alignment Across Latent Spaces for 3D Controllable Face Image SynthesisFeifan Xu, Rui Li, Si Wu, Yong Xu et al.CVPR 2024
- Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent RepresentationZibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng et al.NeurIPS 2023 · 279 citations
- A Unified 3D Human Motion Synthesis Model via Conditional Variational Auto-Encoder∗Yujun Cai, Yiwei Wang, Yiheng Zhu, Tat-Jen Cham et al.ICCV 2021 · 83 citations
- Geometry Aligned Variational Transformer for Image-conditioned Layout GenerationYunning Cao, Ye Ma, Min Zhou, Chuanbin Liu et al.ACM MM 2022 · 37 citations
- IMAGPose: A Unified Conditional Framework for Pose-Guided Person GenerationFei Shen, Jinhui TangNeurIPS 2024 · 172 citations
