In-context Mixing (ICM): Code-mixed Prompts for Multilingual LLMs

We introduce a simple and effective prompt ing technique called incontext mixing (ICM) for effective incontext learning (ICL) with multilingual large language models (MLLMs). With ICM, we modify the fewshot examples within ICL prompts to be intrasententially codemixed by randomly swapping content words in the target languages with their English translations. We observe that ICM prompts yield superior performance in NLP tasks such as disfluency correction, grammar error cor rection and text simplification that demand a close correspondence between the input and out put sequences. Significant improvements are observed mainly for lowresource languages that are underrepresented during the pretrain ing and finetuning of MLLMs. We present an extensive set of experiments to analyze when ICM is effective and what design choices con tribute towards its effectiveness. ICM works consistently and significantly better than other prompting techniques across models of varying capacity such as mT0XXL, BloomZ and GPT 4. Code, prompts and datasets are available here.