Lune

ICCV2023Top-tier venue

Adding Conditional Control to Text-to-Image Diffusion Models

Lvmin Zhang, Anyi Rao, Maneesh Agrawala

Abstract

We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, e.g., edges, depth, segmentation, human pose, etc., with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2,480 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Diffusion Models Beat GANs on Image Synthesis
  • High-Resolution Image Synthesis with Latent Diffusion Models cites Zero-Shot Text-to-Image Generation
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Diffusion Models Beat GANs on Image Synthesis
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Zero-Shot Text-to-Image Generation
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites Zero-Shot Text-to-Image Generation
  • T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites Learning Transferable Visual Models From Natural Language Supervision
  • T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • Diffusion Models Beat GANs on Image Synthesis cites Zero-Shot Text-to-Image Generation
  • Diffusion Models Beat GANs on Image Synthesis cites Learning Transferable Visual Models From Natural Language Supervision
  • Zero-Shot Text-to-Image Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Depth Anything V2 cites High-Resolution Image Synthesis with Latent Diffusion Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Diffusion Models Beat GANs on Image Synthesis
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Zero-Shot Text-to-Image Generation
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Diffusion Models Beat GANs on Image Synthesis
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Zero-Shot Text-to-Image Generation
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2,480

  1. SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations
  2. Depth Anything V2Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao et al.NeurIPS 2024 · 2,305 citations
  3. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging FaceYongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li et al.NeurIPS 2023 · 1,778 citations
  4. T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion ModelsChong Mou, Xintao Wang, Liangbin Xie, Yanze Wu et al.AAAI 2024 · 1,641 citations
  5. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific TuningYuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang et al.ICLR 2024 · 1,493 citations

Builds on38

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Diffusion Models Beat GANs on Image SynthesisPrafulla Dhariwal, Alexander Quinn NicholNeurIPS 2021 · 13,211 citations
  3. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  4. Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
  5. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations

Related papers

  1. CtrLoRA: An Extensible and Efficient Framework for Controllable Image GenerationYifeng Xu, Zhenliang He, Shiguang Shan, Xilin ChenICLR 2025
  2. Modulating Pretrained Diffusion Models for Multimodal Image SynthesisCusuh Ham, James Hays, Jingwan Lu, Krishna Kumar Singh et al.SIGGRAPH 2023 · 17 citations
  3. Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion ModelHan Lin, Jaemin Cho, Abhay Zala, Mohit BansalICLR 2025
  4. DivControl: Knowledge Diversion for Controllable Image GenerationYucheng Xie, Fu Feng, Ruixiao Shi, Jing Wang et al.AAAI 2026 · 4 citations
  5. ControlAR: Controllable Image Generation with Autoregressive ModelsZongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines