Lune

ACL2025Top-tier venue

Multimodal Pragmatic Jailbreak on Text-to-image Models

Tong Liu, Zhixin Lai, Jiawen Wang, Gengyuan Zhang, Shuo Chen, Philip Torr, Vera Demberg, Volker Tresp, Jindong Gu

Abstract

Diffusion models have recently achieved remarkable advancements in terms of image quality and fidelity to textual prompts. Concurrently, the safety of such generative models has become an area of growing concern. This work introduces a novel type of jailbreak, which triggers T2I models to generate the image with visual text, where the image and the text, although considered to be safe in isolation, combine to form unsafe content. To systematically explore this phenomenon, we propose a dataset to evaluate the current diffusion-based text-toimage (T2I) models under such jailbreak. We benchmark nine representative T2I models, including two closed-source commercial models. Experimental results reveal a concerning tendency to produce unsafe content: all tested models suffer from such type of jailbreak, with rates of unsafe generation ranging from around 10% to 70% where DALL•E 3 demonstrates almost the highest unsafety. In real-world scenarios, various filters such as keyword blocklists, customized prompt filters, and NSFW image filters, are commonly employed to mitigate these risks. We evaluate the effectiveness of such filters against our jailbreak and found that, while these filters may be effective for single modality detection, they fail to work against our jailbreak. We also investigate the underlying reason for such jailbreaks, from the perspective of text rendering capability and training data. Our work provides a foundation for further development towards more secure and reliable T2I models. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites Language Models are Few-Shot Learners
  • Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation Models cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Iterative Prompt Refinement for Safer Text-to-Image Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Safe Autoregressive Image Generation with Iterative Self-Improving CodebooksYunqi Xue, Zhijiang Li, Phil Torr, Jindong GuICML 2026 · 1 citation
  2. Not Just Text: Uncovering Vision Modality Typographic Threats in Image Generation ModelsHao Cheng, Erjia Xiao, Jiayan Yang, Jiahang Cao et al.CVPR 2025
  3. Iterative Prompt Refinement for Safer Text-to-Image GenerationJinwoo Jeon, JunHyeok Oh, Hayeong Lee, Byung-Jun LeeEMNLP 2025

Builds on28

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  4. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  5. Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations

Related papers

  1. ColJailBreak: Collaborative Generation and Editing for Jailbreaking Text-to-Image Deep GenerationYizhuo Ma, Shanmin Pang, Qi Guo, Tianyu Wei et al.NeurIPS 2024 · 22 citations
  2. JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution OptimizationHaolun Zheng, Yu He, Tailun Chen, Shuo Shao et al.CVPR 2026 · 3 citations
  3. SneakyPrompt: Jailbreaking Text-to-image Generative ModelsYuchen Yang, Bo Hui, Haolin Yuan, Neil Gong et al.S&P 2024 · 188 citations
  4. Modifier Unlocked: Jailbreaking Text-to-Image Models Through PromptsShuofeng Liu, Mengyao Ma, Minhui Xue, Guangdong BaiS&P 2025
  5. Chain-of-Jailbreak Attack for Image Generation Models via Step by Step EditingWenxuan Wang, Kuiyi Gao, Youliang Yuan, Jen-tse Huang et al.ACL Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines