Lune

ICLR2025Top-tier venue

Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective

Xiangru Zhu, Penglei Sun, Yaoxian Song, Yanghua Xiao, Zhixu Li, Chengyu Wang, Jun Huang, Bei Yang, Xiaoxiao Xu

Abstract

Accurate interpretation and visualization of human instructions are crucial for text-to-image (T2I) synthesis. However, current models struggle to capture semantic variations from word order changes, and existing evaluations, relying on indirect metrics like text-image similarity, fail to reliably assess these challenges. This often obscures poor performance on complex or uncommon linguistic patterns by the focus on frequent word combinations. To address these deficiencies, we propose a novel metric called SemVarEffect and a benchmark named SemVarBench, designed to evaluate the causality between semantic variations in inputs and outputs in T2I synthesis. Semantic variations are achieved through two types of linguistic permutations, while avoiding easily predictable literal variations. Experiments reveal that the CogView-3-Plus and Ideogram 2 performed the best, achieving a score of 0.2/1. Semantic variations in object relations are less understood than attributes, scoring 0.07/1 compared to 0.17-0.19/1. We found that cross-modal alignment in UNet or Transformers plays a crucial role in handling semantic variations, a factor previously overlooked by a focus on textual encoders. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
  • SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis
  • Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
  • R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation cites High-Resolution Image Synthesis with Latent Diffusion Models
  • R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image GenerationKaijie Chen, Zihao Lin, Zhiyang Xu, Ying Shen et al.EMNLP 2025

Builds on27

  1. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  2. Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
  3. SDXL: Improving Latent Diffusion Models for High-Resolution Image SynthesisDustin Podell, Zion English, Kyle Lacey, Andreas Blattmann et al.ICLR 2024 · 4,569 citations
  4. Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
  5. Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image GenerationYuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana et al.NeurIPS 2023 · 1,192 citations

Related papers

  1. TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question AnsweringYushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang et al.ICCV 2023 · 400 citations
  2. Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human ratingOlivia Wiles, Chuhan Zhang, Isabela Albuquerque, Ivana Kajic et al.ICLR 2025
  3. RefVNLI: Towards Scalable Evaluation of Subject-driven Text-to-image GenerationAviv Slobodkin, Hagai Taitelbaum, Yonatan Bitton, Brian Gordon et al.EMNLP Findings 2025
  4. Who Evaluates the Evaluations? Objectively Scoring Text-to-Image Prompt Coherence Metrics with T2IScoreScore (TS2)Michael Saxon, Fatima Jahara, Mahsa Khoshnoodi, Yujie Lu et al.NeurIPS 2024 · 19 citations
  5. UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and PerceptionXinyang Song, Libin Wang, Weining Wang, Shaozhen Liu et al.AAAI 2026
Dusk over the sea between two cliffs drawn in fine vertical lines