InstanceDiffusion: Instance-Level Control for Image Generation
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, Ishan Misra
Abstract
Image Caption: An image depicting in the morning. A brown cute teddy bear, a purple cute teddy bear, a yellow cute teddy bear, a blue cute teddy bear all standing side by side on a red brick road. The scene should be set in front of Pink Castle with clear blue sky overhead, punctuated by fluffy white clouds, and trees with green leaves. The pink castle should loom majestically in the background. Instance Captions: 1-4) A brown/purple/yellow/blue teddy bear; 5) a red brick road; 6) Pink Castle; 7-8) green leaves; 9-10) fluffy white clouds Image Caption: Craft an oil painting: Picture a seaside garden drenched in radiant hues of roses, lilies, and lavender, transitioning gracefully into the expansive azure ocean and blue sky. Integrate a weathered, rustic pathway with steps that invite viewers towards the water's edge, complemented by a prominent bouquet of flowers and plants. Instance Captions: 1-19) roses; 20) sky; 21) ocean; 22) pathway with steps; 23) bouquet of flowers; 24) plant; 25-26) plants
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Cited by top-tier papers94
- Expressive Text-to-Image Generation with Rich TextSongwei Ge, Taesung Park, Jun-Yan Zhu, Jia-Bin HuangICCV 2023 · 102 citations
- HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image GenerationBo Cheng, Yuhang Ma, Liebucha Wu, Shanyuan Liu et al.NeurIPS 2024 · 53 citations
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without GuidanceKuan Heng Lin, Sicheng Mo, Ben Klingher, Fangzhou Mu et al.NeurIPS 2024 · 51 citations
- CreatiDesign: A Unified Multi-Conditional Diffusion Transformer for Creative Graphic DesignHui Zhang, Dexiang Hong, Maoke Yang, Yutao Cheng et al.ICLR 2026 · 40 citations
- Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion ModelsZiyi Wu, Yulia Rubanova, Rishabh Kabra, Drew A. Hudson et al.NeurIPS 2024 · 32 citations
Builds on27
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Denoising Diffusion Implicit ModelsJiaming Song, Chenlin Meng, Stefano ErmonICLR 2021 · 11,743 citations
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
Related papers
- UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in RLRui Tian, Mingfei Gao, Haiming Gang, Jiasen Lu et al.CVPR 2026
- Visual Fact Checker: Enabling High-Fidelity Detailed Caption GenerationYunhao Ge, Xiaohui Zeng, Jacob Samuel Huffman, Tsung-Yi Lin et al.CVPR 2024
- Check, Locate, Rectify: A Training-Free Layout Calibration System for Text- to- Image GenerationBiao Gong, Siteng Huang, Yutong Feng, Shiwei Zhang et al.CVPR 2024
- TextCraftor: Your Text Encoder can be Image Quality ControllerYanyu Li, Xian Liu, Anil Kag, Ju Hu et al.CVPR 2024
- Exploring Sparse MoE in GANs for Text-conditioned Image SynthesisJiapeng Zhu, Ceyuan Yang, Kecheng Zheng, Yinghao Xu et al.CVPR 2025
