ACL Findings2026
FrameNet-Cultures: A Benchmark for Evaluating LLMs via Cross-Cultural Frame Semantics
Abstract
Large language models (LLMs) exhibit cultural biases, yet existing benchmarks rely on closed-form, domain-specific questionnaires. We introduce FRAMENET-CULTURES, a benchmark for evaluating cultural alignment in LLMs based on Fillmore-style frame semantics. Using the EveryCulture encyclopedia, we construct a lexicon of 18 cultural frames (e.g., greeting,child-rearing) across 20 countries, treating it as a structured reference for comparison rather than a definitive representation of contemporary societies. For each frame, we prompt five major LLMs—ChatGPT-5, Gemini-2.5-Flash, Mistral-Large, Qwen-3-Max, DeepSeek-V3.2—three times to generate open-ended instantiations, which are manually annotated and binarized. We measure alignment with country- and continent-level profiles using normalized Hamming distance, and validate cultural recognizability through human evaluation of generated dialogues. Under culture-neutral prompting, outputs align most closely with European profiles, followed by Asian and American ones, indicating a consistent cross-model pattern. With culture-specific prompting, models shift toward the target regions, aligning most strongly with Africa for Ethiopia and with Asia for India. FRAMENET-CULTURES is the first open-ended benchmark for cultural alignment relying on frame semantics. Data, prompts, and annotations are publicly available at https://github.com/neda-jamshidi/FrameNet-Cultures.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- CulturePark: Boosting Cross-cultural Understanding in Large Language Models cites Extracting Cultural Commonsense Knowledge at Scale
- Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment cites Investigating Cultural Alignment of Large Language Models
Builds on8
- Extracting Cultural Commonsense Knowledge at ScaleTuan-Phong Nguyen, Simon Razniewski, Aparna S. Varde, Gerhard WeikumWWW 2023 · 102 citations
- Language (Technology) is Power: A Critical Survey of "Bias" in NLPSu Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna M. WallachACL 2020 · 68 citations
- CulturePark: Boosting Cross-cultural Understanding in Large Language ModelsCheng Li, Damien Teney, Linyi Yang, Qingsong Wen et al.NeurIPS 2024 · 34 citations
- Investigating Cultural Alignment of Large Language ModelsBadr AlKhamissi, Muhammad N. ElNokrashy, Mai Alkhamissi, Mona T. DiabACL 2024 · 27 citations
- Evaluating and Improving Cultural Awareness of Reward Models for LLM AlignmentHongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang et al.ICLR 2026 · 4 citations
Related papers
- CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation MetricsShravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser et al.EMNLP Findings 2025 · 2 citations
- MMAC: A Multilingual, Multimodal Alignment Framework for Cultural Grounding EvaluationWeihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu et al.ACL 2026
- Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language ModelsWenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai et al.ACL 2024 · 21 citations
- MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMsRaoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. HedderichEMNLP Findings 2025
- Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge StructuresSougata Saha, Madhur Jindal, Saurabh Kumar Pandey, Mahardika Ihsani et al.ICML 2026
