Lune

ICLR2025Top-tier venue

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Imanol Schlag, Marzieh Fadaee, Sara Hooker

Abstract

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (i.e., multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge-and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed. We release INCLUDE for public use. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE Serving cites Measuring Massive Multitask Language Understanding
  • XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation cites Unsupervised Cross-lingual Representation Learning at Scale
  • Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation cites Measuring Massive Multitask Language Understanding
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • LORAXBENCH: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages cites XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
  • LORAXBENCH: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages cites Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual EvaluationShivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani et al.ACL 2025 · 144 citations
  2. Stratum: System-Hardware Co-Design with Tiered Monolithic 3D-Stackable DRAM for Efficient MoE ServingYue Pan, Zihan Xia, Po-Kai Hsu, Lanxiang Hu et al.MICRO 2025 · 7 citations
  3. Evaluating and Improving Cultural Awareness of Reward Models for LLM AlignmentHongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang et al.ICLR 2026 · 4 citations
  4. TokSuite: Measuring the Impact of Tokenizer Choice on Language Model BehaviorGül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu et al.ICML 2026 · 3 citations
  5. LORAXBENCH: A Multitask, Multilingual Benchmark Suite for 20 Indonesian LanguagesAlham Fikri Aji, Trevor CohnEMNLP 2025 · 2 citations

Builds on30

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual GeneralisationJunjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig et al.ICML 2020 · 1,132 citations
  5. Unsupervised Cross-lingual Representation Learning at ScaleAlexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary et al.ACL 2020 · 539 citations

Related papers

  1. MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model EvaluationWeihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng et al.EMNLP 2025 · 4 citations
  2. XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question AnsweringKeon-Woo Roh, Yeong-Joon Ju, Seong-Whan LeeEMNLP 2025
  3. Multi-lingual Functional Evaluation for Large Language ModelsVictor Ojewale, Inioluwa Deborah Raji, Suresh VenkatasubramanianACL Findings 2026
  4. SEA-HELM: Southeast Asian Holistic Evaluation of Language ModelsYosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui et al.ACL Findings 2025 · 8 citations
  5. M-RewardBench: Evaluating Reward Models in Multilingual SettingsSrishti Gureja, Lester James Validad Miranda, Shayekh Bin Islam, Rishabh Maheshwary et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines