Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text

Code-switching (CSW) is the act of alternating between two or more languages within a single discourse. This phenomenon is widespread in multilingual communities, and increasingly prevalent in online content, where users naturally mix languages in everyday communication. As a result, Large Language Models (LLMs), now central to content processing and generation, are frequently exposed to code-switched inputs. Given their widespread use, it is crucial to understand how LLMs process and reason about such mixed-language text. This paper presents a systematic evaluation of LLM comprehension under codeswitching by generating CSW variants of established reasoning and comprehension benchmarks. While degradation is evident when foreign tokens disrupt English text-even under linguistic constraints-embedding English into other languages often improves comprehension. Though prompting yields mixed results, finetuning offers a more stable path to degradation mitigation. : (D) : Hume says that beauty is _____. : Hume says that ‫اﻟﺠﻤﺎل‬ is _____. : Hume says that la beauté is _____. : Hume says that Schönheit is _____. : Hume says that 美 is _____. (A) a quality in things themselves (B) a matter of a priori knowledge (C) judged by logical standards (D) no quality in things themselves : (A) : (D) : (C) : (B)