  Accurate alignment between languages is fundamental for improving
cross-lingual pre-trained language models (XLMs). Motivated by the natural
phenomenon of code-switching (CS) in multilingual speakers, CS has been used as
an effective data augmentation method that offers language alignment at the
word- or phrase-level, in contrast to sentence-level via parallel instances.
Existing approaches either use dictionaries or parallel sentences with word
alignment to generate CS data by randomly switching words in a sentence.
However, such methods can be suboptimal as dictionaries disregard semantics,
and syntax might become invalid after random word switching. In this work, we
propose EntityCS, a method that focuses on Entity-level Code-Switching to
capture fine-grained cross-lingual semantics without corrupting syntax. We use
Wikidata and English Wikipedia to construct an entity-centric CS corpus by
switching entities to their counterparts in other languages. We further propose
entity-oriented masking strategies during intermediate model training on the
EntityCS corpus for improving entity prediction. Evaluation of the trained
models on four entity-centric downstream tasks shows consistent improvements
over the baseline with a notable increase of 10% in Fact Retrieval. We release
the corpus and models to assist research on code-switching and enriching XLMs
with external knowledge.
