Toward the Limitation of Code-Switching in Cross-Lingual Transfer

Multilingual pretrained models have shown strong cross-lingual transfer ability. Some works used code-switching sentences, which consist of tokens from multiple languages, to enhance the cross-lingual representation further, and have shown success in many zero-shot cross-lingual tasks. However, code-switched tokens are likely to cause grammatical incoherence in newly substituted sentences, and negatively affect the performance on tokensensitive tasks, such as Part-of-Speech (POS) tagging and Named-Entity-Recognition (NER). This paper mitigates the limitation of the codeswitching method by not only making the token replacement but considering the similarity between the context and the switched tokens so that the newly substituted sentences are grammatically consistent during both training and inference. We conduct experiments on crosslingual POS and NER over 30+ languages, and demonstrate the effectiveness of our method by outperforming the mBERT by 0.95 and original code-switching method by 1.67 on F1 scores. How do menschen look at and अनु भव 艺术 ? How do people look at and experience art ? How do people look at and experience art ? How do Menschen look at and अनु भव 艺术 ? (b): Ours (a): Code-Switched Training Sentence SCONJ AUX NOUN VERB ADP CCPNJ VERB NOUN PUNCT