Stereotype Bias in a Bilingual Setting: A Culturally Grounded Evaluation in Kazakhstan

Stereotype bias in language models has been widely examined in English, but remains largely understudied in bilingual contexts where multiple linguistic and cultural systems interact. This gap is especially important in regions where language use reflects complex historical and sociopolitical influences. In this work, we focus on Kazakhstan, a bilingual society where Kazakh, a low-resource Turkic language, and Russian, a high-resource Slavic language, are both actively used and frequently code-switched in everyday communication. We introduce Aqbileq 1 , a high-quality, humanverified dataset consisting of 5,634 stereotypebearing statements in Kazakh, Russian, and code-switched forms, covering six culturally salient domains. We evaluate both multilingual and Kazakh-specific language models using perplexity-based scoring and pretraining simulations, and find that stereotype bias is most pronounced in code-switched inputs. Our results highlight the limitations of existing evaluation frameworks and emphasize the need for culturally grounded, linguistically inclusive benchmarks to better assess and mitigate bias in language models. Warning: this paper contains example data that may be offensive, harmful, or biased.