EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching

Accurate alignment between languages is fundamental for improving cross-lingual pretrained language models (XLMs).Motivated by the natural phenomenon of code-switching (CS) in multilingual speakers, CS has been used as an effective data augmentation method that offers language alignment at word-or phraselevel, in contrast to sentence-level via parallel instances.Existing approaches either use dictionaries or parallel sentences with wordalignment to generate CS data by randomly switching words in a sentence.However, such methods can be suboptimal as dictionaries disregard semantics, and syntax might become invalid after random word switching.In this work, we propose ENTITYCS, a method that focuses on ENTITY-level Code-Switching to capture fine-grained cross-lingual semantics without corrupting syntax.We use Wikidata and the English Wikipedia to construct an entitycentric CS corpus by switching entities to their counterparts in other languages.We further propose entity-oriented masking strategies during intermediate model training on the ENTI-TYCS corpus for improving entity prediction.Evaluation of the trained models on four entitycentric downstream tasks shows consistent improvements over the baseline with a notable increase of 10% in Fact Retrieval.We release the corpus and models to assist research on codeswitching and enriching XLMs with external knowledge 1 .STATISTIC COUNT Languages 93 English Sentences 54,469,214 English Entities 104,593,076 Average Sentence Length 23.37 Average Entities per Sentence 2 CS Sentences per EN Sentence ≤ 5 CS Sentences 231,124,422 CS Entities 420,907,878A 3.5 8.2 4.7 4.4 6.5 5.3 4.6 2.5 3.1 5.1 8.5 6.3 2.7 2.3 0.9 0.1 1.4 1.2 2.8 3.7 0.2 1.9 0.1 S 9.4 15.2 11.3 11.0 13.4 14.4 11.9 12.3 4.0 16.7 14.2 27.3 19.5 9.2 2.2 0.0 1.7 1.3 5.1 5.6 5.8 3.7 0.4 M 2.1 3.3 2.3 2.6 3.3 3.8 4.5 2.2 2.5 2.6 5.1 2.9 1.1 2.1 0.2 0.1 1.0 1.1 1.4 1.9 0.0 1.6 0.0 CONF A 3.3 4.4 2.9 2.7 4.3 5.5 5.3 3.0 3.0 5.6 9.5 7.3 3.4 4.4 0.9 0.1 1.2 1.1 2.3 2.9 0.6 1.8 0.5 S 7.5 5.2 4.4 3.6 4.9 14.2 11.8 11.4 3.9 15.9 12.6 25.6 18.9 8.8 2.0 0.0 1.4 1.4 4.4 4.3 5.8 3.5 0.5 M 2.6 3.9 2.3 2.7 4.2 4.1 5.2 2.7 2.4 3.4 7.0 4.3 2.07 4.2 0.3 0.1 1.0 1.1 1.3 1.9 0.4 1.5 0.5 MLM 39 IND A 2.3 2.1 3.7 2.9 3.9 2.9 1.9 3.4 1.2 5.0 4.6 4.2 3.6 0.3 0.7 0.0 2.1 1.0 1.4 5.2 0.0 0.0 0.1 S 6.4 5.1 8.7 6.4 9.4 6.0 8.3 8.7 3.1 16.6 9.1 19.3 17.9 2.5 1.8 0.6 2.9 1.1 4.4 8.3 0.3 0.5 0.1 M 1.3 0.9 2.0 2.0 1.9 2.0 1.8 1.9 0.6 2.3 2.4 2.8 2.1 0.2 0.4 0.0 1.8 1.0 0.5 2.2 0.0 0.0 0.1 CONF A 2.5 2.5 3.6 2.9 4.3 2.6 2.0 4.8 1.1 5.7 6.3 5.2 4.2 0.4 0.6 0.1 2.0 1.0 1.2 5.2 0.0 0.0 0.1 S 5.9 4.9 7.6 5.9 9.0 4.4 7.6 7.4 2.5 16.1 8.5 17.2 16.7 2.5 1.6 0.6 2.8 1.1 3.9 7.8 0.3 0.5 0.0 M 1.7 1.8 2.3 2.2 2.6 2.3 1.9 3.4 0.5 3.4 4.6 4.2 2.9 0.3 0.4 0.0 1.7 1.0 0.4 2.4 0.0 0.0 0.1 WEP EN IND A 3.3 18.2 6.1 6.0 5.8 1.1 0.4 0.4 1.1 0.5 8.0 3.5 0.4 0.6 3.7 0.0 3.5 0.6 5.0 4.2 0.1 1.7 1.6 S 8.5 38.3 16.4 18.7 14.9 4.4 3.4 1.4 5.6 2.7 16.8 7.3 4.1 2.5 8.5 0.0 6.9 2.6 9.7 10.3 0.0 6.9 5.4 M 1.6 9.4 2.7 2.9 2.9 0.6 0.3 0.4 0.3 0.3 3.5 1.2 0.1 0.5 2.6 0.0 1.5 0.3 1.5 1.9 0.1 1.1 0.5 CONF A 3.1 16.2 6.4 5.6 5.4 1.1 0.3 0.4 1.1 0.5 7.6 3.4 0.4 0.6 3.6 0.0 3.4 0.5 4.5 4.1 0.1 1.3 1.7 S 7.9 35.8 15.9 17.2 13.3 4.5 2.7 1.6 5.4 2.4 15.8 7.3 4.1 2.5 8.2 0.0 6.6 2.5 8.6 7.8 0.0 6.4 5.4 M 1.5 7.5 3.3 2.9 2.9 0.6 0.3 0.4 0.2 0.3 3.6 1.1 0.1 0.5 2.6 0.0 1.5 0.2 1.5 2.0 0.1 1.0 0.6 39 IND A 6.1 15.6 9.1 11.5 10.5 2.8 6.7 3.7 3.2 6.7 13.2 7.9 4.0 4.6 6.7 0.9 4.3 2.1 7.4 7.2 0.0 2.3 3.3 S 19.4 36.4 24.1 30.3 25.6 14.3 18.5 34.7 12.2 31.5 23.4 36.0 29.8 17.8 18.5 6.1 8.5 5.0 16.9 21.3 0.0 5.4 9.3 M 3.0 7.2 3.9 4.9 4.6 1.5 6.3 2.6 1.0 2.9 7.3 3.9 1.5 4.1 2.5 0.0 2.2 1.4 1.8 3.3 0.0 1.4 0.6 CONF A 4.9 12.1 8.2 9.6 8.8 2.4 3.1 3.3 2.9 5.9 9.3 7.4 3.5 1.9 5.6 0.8 4.1 1.7 6.8 5.7 0.0 1.8 3.3 S 17.4 32.6 22.9 26.5 23.4 12.2 16.7 32.4 11.2 28.3 19.3 34.3 27.1 15.9 16.0 5.6 8.2 4.7 14.9 17.2 0.0 5.1 9.2 M 2.1 4.6 3.3 3.6 3.0 1.2 2.6 2.3 0.8 2.7 3.9 3.7 1.4 1.5 1.8 0.0 2.1 1.0 1.9 2.0 0.0 1.0 0.7 93 IND A 5.8 13.9 7.6 10.1 11.2 2.8 7.2 2.9 2.9 5.8 13.6 8.1 4.4 3.2 7.2 0.6 3.1 2.4 6.8 6.6 1.0 2.5 3.2 S 18.5 34.5 20.0 28.9 25.3 14.0 20.1 26.0 13.0 28.6 25.4 35.0 25.6 17.3 18.2 4.9 6.7 7.2 13.6 17.6 11.6 5.7 8.3 M 2.7 6.6 3.0 4.7 5.2 1.3 6.8 2.3 0.9 2.5 7.6 4.3 2.1 2.7 3.1 0.0 1.8 0.9 1.3 1.4 0.2 1.3 0.4 CONF A 4.6 11.3 6.4 8.6 9.1 2.2 2.7 2.5 2.7 4.9 10.5 7.2 3.5 1.8 6.1 0.6 2.8 2.0 6.2 5.1 0.8 1.4 2.5 S 16.3 31.5 18.4 26.3 22.3 11.8 18.0 24.3 12.2 25.5 22.3 31.3 20.8 14.5 16.4 4.4 6.1 6.6 11.6 15.3 8.9 2.4 7.6 M 1.8 4.7 2.1 3.5 3.4 0.9 2.3 2.0 0.8 2.1 4.9 3.5 1.5 1.4 2.2 0.0 1.6 0.7 1.1 0.9 0.1 0.6 0.4 PEP MS 39 IND A 4.7 15.1 6.9 11.0 9.6 5.0 3.8 3.2 2.0 7.3 9.0 5.5 3.0 3.3 1.9 0.2 3.3 1.5 5.9 5.5 0.0 0.7 0.6 S 15.0 35.2 18.6 29.4 22.0 16.7 15.7 19.4 8.7 29.3 19.2 30.2 24.5 19.9 4.6 1.7 6.4 2.5 10.3 12.8 0.0 1.1 1.2 M 2.4 7.1 2.4 4.5 4.2 2.1 3.5 2.6 0.7 3.3 4.4 3.5 0.5 2.7 1.0 0.0 2.0 1.2 2.4 2.8 0.0 0.5 0.4 CONF A 6.0 15.7 8.1 12.5 11.7 5.7 6.9 5.2 2.9 9.2 14.0 6.3 5.1 6.7 3.4 0.4 3.4 1.5 6.4 5.6 0.0 0.5 0.5 S 13.1 31.9 17.1 27.1 19.6 12.1 13.6 17.7 7.6 26.1 16.0 27.1 21.4 16.9 3.9 1.6 5.3 2.2 9.4 9.0 0.0 0.8 1.1 M 4.3 10.0 4.5 7.1 8.6 4.0 6.7 4.7 2.0 6.2 11.8 4.9 3.3 6.4 2.7 0.2 2.4 1.3 3.9 3.8 0.0 0.2 0.3 PEP MS +MLM EN IND A 2.6 16.8 5.0 5.2 4.9 1.5 0.2 0.6 0.2 0.6 6.3 3.0 0.4 0.6 1.0 0.0 1.