AnyCanvas: Potential Field Guidance for Training-Free Spatial Control in Text-to-Image Diffusion
Tianyi Xie, Zhiyuan Yu, kaihong huang, Beilun Wang, Zhaoyang Wang, Dian Shen
Abstract
Diffusion-based text-to-image (T2I) models have demonstrated remarkable advancements in generating high-quality images. However, while real-world applications like product packaging and logo design necessitate synthesis within irregular geometries, existing methods struggle to handle such constraints. Therefore, generating complete pictures that conform to arbitrary-shaped canvas constraints while maintaining semantic integrity remains a significant challenge. To address this, we introduce AnyCanvas, a training-free framework that leverages a Mask-to-Potential Field paradigm to convert binary masks into a differentiable potential field, which guides content to naturally converge within target regions. Extensive experiments demonstrate that AnyCanvas achieves 4.23% higher spatial adherence to user-specified constraints while maintaining 99.45% of the semantic fidelity measured by CLIP score, leading to a superior harmonic mean of spatial and semantic metrics. AnyCanvas also exhibits robust generalizability across different model backbones and versatile spatial control objectives.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Builds on31
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations
Related papers
- Canvas-to-Image: Compositional Image Generation with Multimodal ControlsYusuf Dalva, Guocheng Gordon Qian, Maya Goldenberg, Tsai-Shien Chen et al.SIGGRAPH 2026
- MultiDiffusion: Fusing Diffusion Paths for Controlled Image GenerationOmer Bar-Tal, Lior Yariv, Yaron Lipman, Tali DekelICML 2023 · 575 citations
- BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained DiffusionJinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu et al.ICCV 2023 · 313 citations
- LoCo: Training-Free Layout-to-Image Synthesis with Localized ConstraintsPeiang Zhao, Han Li, Ruiyang Jin, S. Kevin ZhouACM MM 2025 · 2 citations
- Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic ControlDanfeng Li, Hui Zhang, Sheng Wang, Jiacheng Li et al.NeurIPS 2025 · 11 citations
