EMNLP Findings2025
Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs
Kyubyung Chae, Gihoon Kim, Gyuseong Lee, Taesup Kim, Jaejin Lee, Heejin Kim
Abstract
Recent trends in LLMs development clearly show growing interest in the use and application of sovereign LLMs. The global debate over sovereign LLMs highlights the need for governments to develop their LLMs, tailored to their unique socio-cultural and historical contexts. However, there remains a shortage of frameworks and datasets to verify two critical questions: (1) how well these models align with users' socio-cultural backgrounds, and (2) whether they maintain safety and technical robustness without exposing users to potential harms and risks. To address this gap, we construct a new dataset and introduce an analytic framework for extracting and evaluating the socio-cultural elements of sovereign LLMs, alongside assessments of their technical robustness. Our experimental results demonstrate that while sovereign LLMs play a meaningful role in supporting low-resource languages, they do not always meet the popular claim that these models serve their target users well. We also show that pursuing this untested claim may lead to underestimating critical quality attributes such as safety. Our study suggests that advancing sovereign LLMs requires a more extensive evaluation that incorporates a broader range of wellgrounded and practical criteria.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on21
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Can Large Language Models Be an Alternative to Human Evaluations?David Cheng-Han Chiang, Hung-yi LeeACL 2023 · 254 citations
- CMMLU: Measuring massive multitask language understanding in ChineseHaonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang et al.ACL Findings 2024 · 105 citations
- Evaluating Open-Domain Question Answering in the Era of Large Language ModelsEhsan Kamalloo, Nouha Dziri, Charles L. A. Clarke, Davood RafieiACL 2023 · 96 citations
- A Comprehensive Study of Jailbreak Attack versus Defense for Large Language ModelsZihao Xu, Yi Liu, Gelei Deng, Yuekang Li et al.ACL Findings 2024 · 69 citations
Related papers
- Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional FrameworkMeng-Chen Wu, Si-Chi Chin, Tess Wood, Ayush Goyal et al.EMNLP 2025
- Qorǵau: Evaluating Safety in Kazakh-Russian Bilingual ContextsMaiya Goloburda, Nurkhan Laiyk, Diana Turmakhan, Yuxia Wang et al.ACL Findings 2025 · 6 citations
- Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety AlignmentSomnath Banerjee, Sayan Layek, Pratyush Chatterjee, Animesh Mukherjee et al.EMNLP Findings 2025 · 1 citation
- Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare QueriesYiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu et al.WWW 2024 · 126 citations
- SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful RequestsPunya Syon Pandey, Hai Son Le, Devansh Bhardwaj, Zhijing JinICLR 2026 · 7 citations
