Lune

ACL Findings2024

KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge

Jiyoung Lee, Minwoo Kim, Seungho Kim, Junghwan Kim, Seunghyun Won, Hwaran Lee, Edward Choi

Abstract

For Large Language Models (LLMs) to be effectively deployed in a specific country, they must possess an understanding of the nation's culture and basic knowledge. To this end, we introduce National Alignment, which measures an alignment between an LLM and a targeted country from two aspects: social value alignment and common knowledge alignment. Social value alignment evaluates how well the model understands nation-specific social values, while common knowledge alignment examines how well the model captures basic knowledge related to the nation. We constructed KorNAT, the first benchmark that measures national alignment with South Korea. For the social value dataset, we obtained ground truth labels from a large-scale survey involving 6,174 unique Korean participants. For the common knowledge dataset, we constructed samples based on Korean textbooks and GED reference materials. KorNAT contains 4K and 6K multiplechoice questions for social value and common knowledge, respectively. Our dataset creation process is meticulously designed and based on statistical sampling theory and was refined through multiple rounds of human review. The experiment results of seven LLMs reveal that only a few models met our reference score, indicating a potential for further enhancement. KorNAT has received government approval after passing an assessment conducted by a government-affiliated organization dedicated to evaluating dataset quality. Samples and detailed evaluation protocols of our dataset can be found in https://huggingface.co/ datasets/datumo/KorNAT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • Whose Opinions Do Language Models Reflect? cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Culture is Not Trivia: Sociocultural Theory for Cultural NLPNaitian Zhou, David Bamman, Isaac L. BleamanACL 2025 · 33 citations
  2. NormGenesis: Multicultural Dialogue Generation via Exemplar-Guided Social Norm Modeling and Violation RecoveryMinki Hong, Jangho Choi, Jihie KimEMNLP 2025 · 1 citation
  3. Implicit Values Embedded in How Humans and LLMs Complete Subjective Everyday TasksArjun Arunasalam, Madison Pickering, Z. Berkay Celik, Blase UrEMNLP 2025
  4. NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local CommunitiesAbdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata et al.EMNLP 2025

Builds on13

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  3. Whose Opinions Do Language Models Reflect?Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee et al.ICML 2023 · 764 citations
  4. QASC: A Dataset for Question Answering via Sentence CompositionTushar Khot, Peter Clark, Michal Guerquin, Peter Jansen et al.AAAI 2020 · 387 citations
  5. Process for Adapting Language Models to Society (PALMS) with Values-Targeted DatasetsIrene Solaiman, Christy DennisonNeurIPS 2021 · 276 citations

Related papers

  1. Benchmarking Multi-National Value Alignment for Large Language ModelsChengyi Ju, Weijie Shi, Chengzhong Liu, Jiaming Ji et al.ACL Findings 2025 · 1 citation
  2. MMAC: A Multilingual, Multimodal Alignment Framework for Cultural Grounding EvaluationWeihua Zheng, Zhengyuan Liu, Tanmoy Chakraborty, Weiwen Xu et al.ACL 2026
  3. Incorporating Diverse Perspectives in Cultural Alignment: Survey of Evaluation Benchmarks Through A Three-Dimensional FrameworkMeng-Chen Wu, Si-Chi Chin, Tess Wood, Ayush Goyal et al.EMNLP 2025
  4. KoCommonGEN v2: A Benchmark for Navigating Korean Commonsense Reasoning Challenges in Large Language ModelsJaehyung Seo, Jaewook Lee, Chanjun Park, Seongtae Hong et al.ACL Findings 2024 · 4 citations
  5. AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value DifferenceJing Yao, Shitong Duan, Xiaoyuan Yi, Dongkuan Xu et al.ICLR 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines