Lune

ACL Findings2025

SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture

Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka, Sriparna Saha

Abstract

Language models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising of 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture namely rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SAN-SKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Training language models to follow instructions with human feedback cites Language (Technology) is Power: A Critical Survey of "Bias" in NLP
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • Common to Whom? Regional Cultural Commonsense and LLM Bias in India cites Language Models are Few-Shot Learners
  • TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories cites Investigating Cultural Alignment of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL ConferencesYusuke Sakai, Hidetaka Kamigaito, Taro WatanabeACL 2026 · 18 citations
  2. CLINIC : Evaluating Multilingual Trustworthiness in Language Models for HealthcareAkash Ghosh, Srivarshinee Sridhar, Raghav Kaushik Ravi, Muhsin Muhsin et al.ICML 2026 · 6 citations
  3. Common to Whom? Regional Cultural Commonsense and LLM Bias in IndiaSangmitra Madhusudan, Trush Shashank More, Steph Buongiorno, Renata Dividino et al.ACL 2026 · 1 citation
  4. TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated StoriesKirti Bhagat, Shaily Bhatt, Athul Velagapudi, Aditya Vashistha et al.CHI 2026 · 1 citation

Builds on9

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Language (Technology) is Power: A Critical Survey of "Bias" in NLPSu Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna M. WallachACL 2020 · 68 citations
  4. Investigating Cultural Alignment of Large Language ModelsBadr AlKhamissi, Muhammad N. ElNokrashy, Mai Alkhamissi, Mona T. DiabACL 2024 · 27 citations
  5. Ready Player One! Eliciting Diverse Knowledge Using A Configurable GameAgathe Balayn, Gaole He, Andrea Hu, Jie Yang et al.WWW 2022 · 15 citations

Related papers

  1. DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian ContextPramit Sahoo, Maharaj Brahma, Maunendra Sankar DesarkarEMNLP 2025
  2. DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian CultureArijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka et al.EMNLP 2025 · 1 citation
  3. IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic LanguagesHarman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari et al.ACL 2024
  4. Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot SettingsHamna, Gayatri Bhat, Sourabrata Mukherjee, Faisal M. Lalani et al.CHI 2026 · 1 citation
  5. IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMsAli Faraz, Akash, Shaharukh Khan, Raja Kolla et al.ICLR 2026 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines