EMNLP Findings2025
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd van Steenkiste, Yash Goyal, Karolina Stanczak, Aishwarya Agrawal
Abstract
The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts -where missed cues can stereotype communities and undermine usability. In this work, we present the first study to systematically quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) as well as implicit (unstated, implied by the prompt's cultural context) cultural expectations. To this end, we introduce CULTURALFRAMES, a novel benchmark designed for rigorous human evaluation of cultural representation in visual generations. Spanning 10 countries and 5 socio-cultural domains, CULTURALFRAMES comprises 983 prompts, 3,637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. We find that across models and countries, cultural expectations are missed an average of 44% of the time. Among these failures, explicit expectations are missed at a surprisingly high average rate of 68%, while implicit expectation failures are also significant, averaging 49%. Furthermore, we show that existing T2I evaluation metrics correlate poorly with human judgments of cultural alignment, irrespective of their internal reasoning. Collectively, our findings expose critical gaps, provide a concrete testbed, and outline actionable directions for developing culturally informed T2I models and metrics that improve global usability.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation cites ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation cites Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
- Culture in Action: Evaluating Text-to-Image Models through Social Activities cites Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
- Culture in Action: Evaluating Text-to-Image Models through Social Activities cites ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
- Culture in Action: Evaluating Text-to-Image Models through Social Activities cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Culture in Action: Evaluating Text-to-Image Models through Social Activities cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- Synthetic History: Evaluating Visual Representations of the Past in Diffusion Models cites Scaling Rectified Flow Transformers for High-Resolution Image Synthesis
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models cites Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
- Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Cited by top-tier papers5
- Culture in Action: Evaluating Text-to-Image Models through Social ActivitiesSina Malakouti, Boqing Gong, Adriana KovashkaICLR 2026 · 9 citations
- Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to ModelsLeander Girrbach, Stephan Alaniz, Genevieve Smith, Trevor Darrell et al.ICLR 2026 · 5 citations
- Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective ResponsesChongyuan Dai, Yaling Shen, Zihan Gao, Jia Li et al.ACL 2026 · 4 citations
- Synthetic History: Evaluating Visual Representations of the Past in Diffusion ModelsMaria-Teresa De Rosa Palmini, Eva CetinicICLR 2026 · 1 citation
- Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety AlignmentArkadiy Saakyan, Charvi Rastogi, Lora AroyoICML 2026
Builds on22
- Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations
- Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
- ImageReward: Learning and Evaluating Human Preferences for Text-to-Image GenerationJiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong et al.NeurIPS 2023 · 1,310 citations
- Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image GenerationYuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana et al.NeurIPS 2023 · 1,192 citations
- CLIPScore: A Reference-free Evaluation Metric for Image CaptioningJack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras et al.EMNLP 2021 · 937 citations
Related papers
- Diffusion Models Through a Global Lens: Are They Culturally Inclusive?Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad et al.ACL 2025
- FrameNet-Cultures: A Benchmark for Evaluating LLMs via Cross-Cultural Frame SemanticsNeda Jamshidi, Anders Søgaard, Monica BianchiniACL Findings 2026
- Partiality and Misconception: Investigating Cultural Representativeness in Text-to-Image ModelsLili Zhang, Xi Liao, Zaijia Yang, Baihang Gao et al.CHI 2024 · 17 citations
- WISE: World Knowledge-Informed Semantic Evaluation for Text-to-Image GenerationYuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin et al.ICML 2026 · 195 citations
- YinYang-Align: A new Benchmark for Competing Objectives and Introducing Multi-Objective Preference based Text-to-Image AlignmentAmitava Das, Yaswanth Narsupalli, Gurpreet Singh, Vinija Jain et al.ACL Findings 2025
