Lune

ACL Findings2026

FrameNet-Cultures: A Benchmark for Evaluating LLMs via Cross-Cultural Frame Semantics

Neda Jamshidi, Anders Søgaard, Monica Bianchini

Abstract

Large language models (LLMs) exhibit cultural biases, yet existing benchmarks rely on closed-form, domain-specific questionnaires. We introduce FRAMENET-CULTURES, a benchmark for evaluating cultural alignment in LLMs based on Fillmore-style frame semantics. Using the EveryCulture encyclopedia, we construct a lexicon of 18 cultural frames (e.g., greeting,child-rearing) across 20 countries, treating it as a structured reference for comparison rather than a definitive representation of contemporary societies. For each frame, we prompt five major LLMs—ChatGPT-5, Gemini-2.5-Flash, Mistral-Large, Qwen-3-Max, DeepSeek-V3.2—three times to generate open-ended instantiations, which are manually annotated and binarized. We measure alignment with country- and continent-level profiles using normalized Hamming distance, and validate cultural recognizability through human evaluation of generated dialogues. Under culture-neutral prompting, outputs align most closely with European profiles, followed by Asian and American ones, indicating a consistent cross-model pattern. With culture-specific prompting, models shift toward the target regions, aligning most strongly with Africa for Ethiopia and with Asia for India. FRAMENET-CULTURES is the first open-ended benchmark for cultural alignment relying on frame semantics. Data, prompts, and annotations are publicly available at https://github.com/neda-jamshidi/FrameNet-Cultures.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CulturePark: Boosting Cross-cultural Understanding in Large Language Models cites Extracting Cultural Commonsense Knowledge at Scale
  • Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment cites Investigating Cultural Alignment of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on8

  1. Extracting Cultural Commonsense Knowledge at ScaleTuan-Phong Nguyen, Simon Razniewski, Aparna S. Varde, Gerhard WeikumWWW 2023 · 102 citations
  2. Language (Technology) is Power: A Critical Survey of "Bias" in NLPSu Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna M. WallachACL 2020 · 68 citations
  3. CulturePark: Boosting Cross-cultural Understanding in Large Language ModelsCheng Li, Damien Teney, Linyi Yang, Qingsong Wen et al.NeurIPS 2024 · 34 citations
  4. Investigating Cultural Alignment of Large Language ModelsBadr AlKhamissi, Muhammad N. ElNokrashy, Mai Alkhamissi, Mona T. DiabACL 2024 · 27 citations
  5. Evaluating and Improving Cultural Awareness of Reward Models for LLM AlignmentHongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang et al.ICLR 2026 · 4 citations

Related papers

  1. CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation MetricsShravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser et al.EMNLP Findings 2025 · 2 citations
  2. MMAC: A Multilingual, Multimodal Alignment Framework for Cultural Grounding EvaluationWeihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu et al.ACL 2026
  3. Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language ModelsWenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai et al.ACL 2024 · 21 citations
  4. MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMsRaoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. HedderichEMNLP Findings 2025
  5. Measuring Meta-Cultural Competency: A Spectral Framework for LLM Knowledge StructuresSougata Saha, Madhur Jindal, Saurabh Kumar Pandey, Mahardika Ihsani et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines