Lune

ACL Findings2024

DiffChat: Learning to Chat with Text-to-Image Synthesis Models for Interactive Image Creation

Jiapeng Wang, Chengyu Wang, Tingfeng Cao, Jun Huang, Lianwen Jin

Abstract

We present DiffChat, a novel method to align Large Language Models (LLMs) to "chat" with prompt-as-input Text-to-Image Synthesis (TIS) models (e.g., Stable Diffusion) for interactive image creation. Given a raw prompt/image and a user-specified instruction, DiffChat can effectively make appropriate modifications and generate the target prompt, which can be leveraged to create the target image of high quality. To achieve this, we first collect an instructionfollowing prompt engineering dataset named InstructPE for the supervised training of Dif-fChat. Next, we propose a reinforcement learning framework with the feedback of three core criteria for image creation, i.e., aesthetics, user preference and content integrity. It involves an action-space dynamic modification technique to obtain more relevant positive samples and harder negative samples during the off-policy sampling. Content integrity is also introduced into the value estimation function for further improvement of produced images. Our method can exhibit superior performance than baseline models and strong competitors based on both automatic and human evaluations, which fully demonstrates its effectiveness. 1 * Contribution during internship at Alibaba Group.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • High-Resolution Image Synthesis with Latent Diffusion Models cites Denoising Diffusion Probabilistic Models
  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites Language Models are Few-Shot Learners
  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites Twin Co-Adaptive Dialogue for Progressive Image Generation
  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites Denoising Diffusion Probabilistic Models
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Language Models are Few-Shot Learners
  • Twin Co-Adaptive Dialogue for Progressive Image Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting cites High-Resolution Image Synthesis with Latent Diffusion Models
  • ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting cites Language Models are Few-Shot Learners
  • ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting cites Denoising Diffusion Probabilistic Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Twin Co-Adaptive Dialogue for Progressive Image GenerationJianhui Wang, Yangfan He, Yan Zhong, Xinyuan Song et al.ACM MM 2025 · 4 citations
  2. Safe Autoregressive Image Generation with Iterative Self-Improving CodebooksYunqi Xue, Zhijiang Li, Phil Torr, Jindong GuICML 2026 · 1 citation
  3. ChatGen: Automatic Text-to-Image Generation From FreeStyle ChattingChengyou Jia, Changliang Xia, Zhuohang Dang, Weijia Wu et al.CVPR 2025

Builds on16

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  4. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  5. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations

Related papers

  1. Optimizing Prompts for Text-to-Image GenerationYaru Hao, Zewen Chi, Li Dong, Furu WeiNeurIPS 2023 · 303 citations
  2. Diffusion Model Alignment Using Direct Preference OptimizationBram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou et al.CVPR 2024 · 89 citations
  3. Rethinking Direct Preference Optimization in Diffusion ModelsJunyong Kang, Seohyun Lim, Kyungjune Baek, Hyunjung ShimAAAI 2026
  4. Dynamic Prompt Optimizing for Text-to-Image GenerationWenyi Mo, Tianyu Zhang, Yalong Bai, Bing Su et al.CVPR 2024 · 15 citations
  5. CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image GenerationMinghao Fu, Guo-Hua Wang, Liangfu Cao, Qing-Guo Chen et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines